cffi vs. Python C Extension

[davidblewett]

I am curious about the choice to use a Python C extension vs. using cffi? cffi should be easier to maintain, be much more performant on alternative runtimes (PyPy [1]), and core developers advocate for it [2]. There are high profile projects using it (pyzmq for example [3]). The documentation is pretty good [4], and instructions for packaging have gotten much simpler in recent months [5].

1. http://doc.pypy.org/en/latest/faq.html#see-below
2. http://www.snarky.ca/try-to-not-use-the-c-api-directly
3. https://github.com/zeromq/pyzmq
4. http://cffi.readthedocs.io/en/latest/overview.html
5. https://caremad.io/posts/2015/06/distributing-a-cffi-project-redux/

[davidblewett]

Just wanted to clarify that I do appreciate the work put in for the current implementation. Our largest use case for Kafka from Python is built on PyPy. That generally limits our options to either pure Python libraries or ones using cffi to wrap underlying C libraries. Would you be interested in something that tried to straddle this divide similar to how pyzmq offers both a Cython and a cffi backend?

[jeffwidman]

Also curious to know the answer here.

[edenhill]

Our experience, and also input from the Pykafka folks, is that CFFI performance was going to be a problem.

It is unlikely that we'll provide a dual stack implementation any time soon,
if you are a customer, feel free to raise through support and this will help us prioritize.