Building on Linux

[rafael-telles]

I am having some problems to run pip install confluent-kafka or python setup.py build, I already have installed common build dependencies (like librdkafka-dev librdkafka1 build-essential autoconf libtool python-dev)

Here is the log:

▶ python setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/confluent_kafka
copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka
creating build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/confluent_kafka
creating build/temp.linux-x86_64-2.7/confluent_kafka/src
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:0:
confluent_kafka/src/confluent_kafka.h:114:1: error: unknown type name ‘rd_kafka_topic_partition_list_t’
 PyObject *c_parts_to_py (const rd_kafka_topic_partition_list_t *c_parts);
 ^
confluent_kafka/src/confluent_kafka.h:115:1: error: unknown type name ‘rd_kafka_topic_partition_list_t’
 rd_kafka_topic_partition_list_t *py_to_c_parts (PyObject *plist);
 ^
confluent_kafka/src/confluent_kafka.c: In function ‘KafkaError_name’:
confluent_kafka/src/confluent_kafka.c:74:2: warning: implicit declaration of function ‘rd_kafka_err2name’ [-Wimplicit-function-declaration]
  return cfl_PyUnistr(_FromString(rd_kafka_err2name(self->code)));
  ^
confluent_kafka/src/confluent_kafka.c:74:2: warning: passing argument 1 of ‘PyUnicodeUCS4_FromString’ makes pointer from integer without a cast [enabled by default]
In file included from /usr/include/python2.7/Python.h:85:0,
                 from confluent_kafka/src/confluent_kafka.h:17,
                 from confluent_kafka/src/confluent_kafka.c:17:
/usr/include/python2.7/unicodeobject.h:281:31: note: expected ‘const char *’ but argument is of type ‘int’
 # define PyUnicode_FromString PyUnicodeUCS4_FromString
                               ^
/usr/include/python2.7/unicodeobject.h:477:23: note: in expansion of macro ‘PyUnicode_FromString’
 PyAPI_FUNC(PyObject*) PyUnicode_FromString(
                       ^
confluent_kafka/src/confluent_kafka.c: In function ‘TopicPartition_new’:
confluent_kafka/src/confluent_kafka.c:551:21: error: ‘RD_KAFKA_OFFSET_INVALID’ undeclared (first use in this function)
  long long offset = RD_KAFKA_OFFSET_INVALID;
                     ^
confluent_kafka/src/confluent_kafka.c:551:21: note: each undeclared identifier is reported only once for each function it appears in
confluent_kafka/src/confluent_kafka.c: At top level:
confluent_kafka/src/confluent_kafka.c:746:1: error: unknown type name ‘rd_kafka_topic_partition_list_t’
 PyObject *c_parts_to_py (const rd_kafka_topic_partition_list_t *c_parts) {
 ^
confluent_kafka/src/confluent_kafka.c: In function ‘c_parts_to_py’:
confluent_kafka/src/confluent_kafka.c:750:28: error: request for member ‘cnt’ in something not a structure or union
  parts = PyList_New(c_parts->cnt);
                            ^
confluent_kafka/src/confluent_kafka.c:752:26: error: request for member ‘cnt’ in something not a structure or union
  for (i = 0 ; i < c_parts->cnt ; i++) {
                          ^
confluent_kafka/src/confluent_kafka.c:753:3: error: unknown type name ‘rd_kafka_topic_partition_t’
   const rd_kafka_topic_partition_t *rktpar = &c_parts->elems[i];
   ^
confluent_kafka/src/confluent_kafka.c:753:54: error: request for member ‘elems’ in something not a structure or union
   const rd_kafka_topic_partition_t *rktpar = &c_parts->elems[i];
                                                      ^
In file included from /usr/include/python2.7/Python.h:100:0,
                 from confluent_kafka/src/confluent_kafka.h:17,
                 from confluent_kafka/src/confluent_kafka.c:17:
confluent_kafka/src/confluent_kafka.c:756:12: error: request for member ‘topic’ in something not a structure or union
      rktpar->topic, rktpar->partition,
            ^
/usr/include/python2.7/listobject.h:62:74: note: in definition of macro ‘PyList_SET_ITEM’
 #define PyList_SET_ITEM(op, i, v) (((PyListObject *)(op))->ob_item[i] = (v))
                                                                          ^
confluent_kafka/src/confluent_kafka.c:756:27: error: request for member ‘partition’ in something not a structure or union
      rktpar->topic, rktpar->partition,
                           ^
/usr/include/python2.7/listobject.h:62:74: note: in definition of macro ‘PyList_SET_ITEM’
 #define PyList_SET_ITEM(op, i, v) (((PyListObject *)(op))->ob_item[i] = (v))
                                                                          ^
confluent_kafka/src/confluent_kafka.c:757:12: error: request for member ‘offset’ in something not a structure or union
      rktpar->offset, rktpar->err));
            ^
/usr/include/python2.7/listobject.h:62:74: note: in definition of macro ‘PyList_SET_ITEM’
 #define PyList_SET_ITEM(op, i, v) (((PyListObject *)(op))->ob_item[i] = (v))
                                                                          ^
confluent_kafka/src/confluent_kafka.c:757:28: error: request for member ‘err’ in something not a structure or union
      rktpar->offset, rktpar->err));
                            ^
/usr/include/python2.7/listobject.h:62:74: note: in definition of macro ‘PyList_SET_ITEM’
 #define PyList_SET_ITEM(op, i, v) (((PyListObject *)(op))->ob_item[i] = (v))
                                                                          ^
confluent_kafka/src/confluent_kafka.c: At top level:
confluent_kafka/src/confluent_kafka.c:769:1: error: unknown type name ‘rd_kafka_topic_partition_list_t’
 rd_kafka_topic_partition_list_t *py_to_c_parts (PyObject *plist) {
 ^
confluent_kafka/src/confluent_kafka.c: In function ‘py_to_c_parts’:
confluent_kafka/src/confluent_kafka.c:770:2: error: unknown type name ‘rd_kafka_topic_partition_list_t’
  rd_kafka_topic_partition_list_t *c_parts;
  ^
confluent_kafka/src/confluent_kafka.c:779:2: warning: implicit declaration of function ‘rd_kafka_topic_partition_list_new’ [-Wimplicit-function-declaration]
  c_parts = rd_kafka_topic_partition_list_new(PyList_Size(plist));
  ^
confluent_kafka/src/confluent_kafka.c:779:10: warning: assignment makes pointer from integer without a cast [enabled by default]
  c_parts = rd_kafka_topic_partition_list_new(PyList_Size(plist));
          ^
confluent_kafka/src/confluent_kafka.c:790:4: warning: implicit declaration of function ‘rd_kafka_topic_partition_list_destroy’ [-Wimplicit-function-declaration]
    rd_kafka_topic_partition_list_destroy(c_parts);
    ^
confluent_kafka/src/confluent_kafka.c:794:3: warning: implicit declaration of function ‘rd_kafka_topic_partition_list_add’ [-Wimplicit-function-declaration]
   rd_kafka_topic_partition_list_add(c_parts,
   ^
confluent_kafka/src/confluent_kafka.c:796:23: error: invalid type argument of ‘->’ (have ‘int’)
         tp->partition)->offset =
                       ^
confluent_kafka/src/confluent_kafka.c: In function ‘producer_conf_set_special’:
confluent_kafka/src/confluent_kafka.c:915:13: error: ‘rd_kafka_msg_partitioner_consistent’ undeclared (first use in this function)
      tconf, rd_kafka_msg_partitioner_consistent);
             ^
confluent_kafka/src/confluent_kafka.c:918:13: error: ‘rd_kafka_msg_partitioner_consistent_random’ undeclared (first use in this function)
      tconf, rd_kafka_msg_partitioner_consistent_random);
             ^
In file included from confluent_kafka/src/confluent_kafka.c:17:0:
confluent_kafka/src/confluent_kafka.c:945:6: error: ‘RD_KAFKA_RESP_ERR__NOT_IMPLEMENTED’ undeclared (first use in this function)
      RD_KAFKA_RESP_ERR__NOT_IMPLEMENTED,
      ^
confluent_kafka/src/confluent_kafka.h:97:35: note: in definition of macro ‘cfl_PyErr_Format’
   PyObject *_eo = KafkaError_new0(err, __VA_ARGS__); \
                                   ^
confluent_kafka/src/confluent_kafka.c: In function ‘common_conf_setup’:
confluent_kafka/src/confluent_kafka.c:1108:2: warning: implicit declaration of function ‘rd_kafka_conf_set_default_topic_conf’ [-Wimplicit-function-declaration]
  rd_kafka_conf_set_default_topic_conf(conf, tconf);
  ^
confluent_kafka/src/confluent_kafka.c: In function ‘KafkaError_add_errs’:
confluent_kafka/src/confluent_kafka.c:1174:2: warning: implicit declaration of function ‘rd_kafka_get_err_descs’ [-Wimplicit-function-declaration]
  rd_kafka_get_err_descs(&descs, &cnt);
  ^
confluent_kafka/src/confluent_kafka.c:1210:3: error: invalid use of undefined type ‘struct rd_kafka_err_desc’
   if (!descs[i].desc)
   ^
confluent_kafka/src/confluent_kafka.c:1210:13: error: dereferencing pointer to incomplete type
   if (!descs[i].desc)
             ^
confluent_kafka/src/confluent_kafka.c:1213:3: error: invalid use of undefined type ‘struct rd_kafka_err_desc’
   code = PyLong_FromLong(descs[i].code);
   ^
confluent_kafka/src/confluent_kafka.c:1213:31: error: dereferencing pointer to incomplete type
   code = PyLong_FromLong(descs[i].code);
                               ^
confluent_kafka/src/confluent_kafka.c:1215:3: error: invalid use of undefined type ‘struct rd_kafka_err_desc’
   PyDict_SetItemString(dict, descs[i].name, code);
   ^
confluent_kafka/src/confluent_kafka.c:1215:35: error: dereferencing pointer to incomplete type
   PyDict_SetItemString(dict, descs[i].name, code);
                                   ^
confluent_kafka/src/confluent_kafka.c:1219:3: error: invalid use of undefined type ‘struct rd_kafka_err_desc’
   _PRINT("| %-*.*s | %-*.*s |\n"
   ^
confluent_kafka/src/confluent_kafka.c:1221:33: error: dereferencing pointer to incomplete type
          _COL1_W, _COL1_W, descs[i].name,
                                 ^
confluent_kafka/src/confluent_kafka.c:1188:43: note: in definition of macro ‘_PRINT’
   _len = snprintf(tmpdoc, sizeof(tmpdoc), __VA_ARGS__); \
                                           ^
confluent_kafka/src/confluent_kafka.c:1219:3: error: invalid use of undefined type ‘struct rd_kafka_err_desc’
   _PRINT("| %-*.*s | %-*.*s |\n"
   ^
confluent_kafka/src/confluent_kafka.c:1222:33: error: dereferencing pointer to incomplete type
          _COL2_W, _COL2_W, descs[i].desc,
                                 ^
confluent_kafka/src/confluent_kafka.c:1188:43: note: in definition of macro ‘_PRINT’
   _len = snprintf(tmpdoc, sizeof(tmpdoc), __VA_ARGS__); \
                                           ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1```

[edenhill]

It looks like your librdkafka is too old, you'll need librdkafka >=0.9.1.
Packages are available from Confluent or Debian