Error, when installing requirements in docker-compose. Macbook M2

[timurchikk]

Description

• Macbook with M2 chip. Getting error in docker-compose, when installing requirements
• confluent-kafka version == 1.9.2
• docker-compose version == 1.29.2
• Docker desktop version == 4.15.0
• If you want more details, i can provide
  ...
• Error:
  Auto Offset Reset for existing Consumer Group #11 41.17 Running setup.py install for confluent-kafka: started
  Auto Offset Reset for existing Consumer Group #11 41.39 Running setup.py install for confluent-kafka: finished with status 'error'
  Auto Offset Reset for existing Consumer Group #11 41.40 error: subprocess-exited-with-error
  Auto Offset Reset for existing Consumer Group #11 41.40
  Auto Offset Reset for existing Consumer Group #11 41.40 × Running setup.py install for confluent-kafka did not run successfully.
  Auto Offset Reset for existing Consumer Group #11 41.40 │ exit code: 1
  Auto Offset Reset for existing Consumer Group #11 41.40 ╰─> [53 lines of output]
  Auto Offset Reset for existing Consumer Group #11 41.40 running install
  Auto Offset Reset for existing Consumer Group #11 41.40 running build
  Auto Offset Reset for existing Consumer Group #11 41.40 running build_py
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/serializing_producer.py -> build/lib.linux-aarch64-3.9/confluent_kafka
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-aarch64-3.9/confluent_kafka
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/error.py -> build/lib.linux-aarch64-3.9/confluent_kafka
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/schema_registry/avro.py -> build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/schema_registry/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/schema_registry/error.py -> build/lib.linux-aarch64-3.9/confluent_kafka/schema_registry
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka/admin
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/admin/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka/admin
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/admin/_acl.py -> build/lib.linux-aarch64-3.9/confluent_kafka/admin
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/admin/_config.py -> build/lib.linux-aarch64-3.9/confluent_kafka/admin
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/admin/_resource.py -> build/lib.linux-aarch64-3.9/confluent_kafka/admin
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka/serialization
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/serialization/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka/serialization
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka/avro
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-aarch64-3.9/confluent_kafka/avro
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/avro/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka/avro
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/avro/load.py -> build/lib.linux-aarch64-3.9/confluent_kafka/avro
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/avro/error.py -> build/lib.linux-aarch64-3.9/confluent_kafka/avro
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka/kafkatest
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/kafkatest/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka/kafkatest
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-aarch64-3.9/confluent_kafka/kafkatest
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-aarch64-3.9/confluent_kafka/kafkatest
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-aarch64-3.9/confluent_kafka/kafkatest
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/lib.linux-aarch64-3.9/confluent_kafka/avro/serializer
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/avro/serializer/init.py -> build/lib.linux-aarch64-3.9/confluent_kafka/avro/serializer
  Auto Offset Reset for existing Consumer Group #11 41.40 copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-aarch64-3.9/confluent_kafka/avro/serializer
  Auto Offset Reset for existing Consumer Group #11 41.40 running build_ext
  Auto Offset Reset for existing Consumer Group #11 41.40 building 'confluent_kafka.cimpl' extension
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9/tmp
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9/tmp/pip-install-er5wkw5e
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9/tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9/tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9/tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src/confluent_kafka
  Auto Offset Reset for existing Consumer Group #11 41.40 creating build/temp.linux-aarch64-3.9/tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src/confluent_kafka/src
  Auto Offset Reset for existing Consumer Group #11 41.40 gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c /tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src/confluent_kafka/src/Admin.c -o build/temp.linux-aarch64-3.9/tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src/confluent_kafka/src/Admin.o
  Auto Offset Reset for existing Consumer Group #11 41.40 In file included from /tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src/confluent_kafka/src/Admin.c:17:
  Auto Offset Reset for existing Consumer Group #11 41.40 /tmp/pip-install-er5wkw5e/confluent-kafka_7c14024e9af544eab5cb0073fbb5e347/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
  Auto Offset Reset for existing Consumer Group #11 41.40 23 | #include <librdkafka/rdkafka.h>
  Auto Offset Reset for existing Consumer Group #11 41.40 | ^~~~~~~~~~~~~~~~~~~~~~
  Auto Offset Reset for existing Consumer Group #11 41.40 compilation terminated.
  Auto Offset Reset for existing Consumer Group #11 41.40 error: command '/usr/bin/gcc' failed with exit code 1
  Auto Offset Reset for existing Consumer Group #11 41.40 [end of output]
  Auto Offset Reset for existing Consumer Group #11 41.40
  Auto Offset Reset for existing Consumer Group #11 41.40 note: This error originates from a subprocess, and is likely not a problem with pip.
  Auto Offset Reset for existing Consumer Group #11 41.40 error: legacy-install-failure
  Auto Offset Reset for existing Consumer Group #11 41.40
  Auto Offset Reset for existing Consumer Group #11 41.40 × Encountered error while trying to install package.
  Auto Offset Reset for existing Consumer Group #11 41.40 ╰─> confluent-kafka
  Auto Offset Reset for existing Consumer Group #11 41.40
  Auto Offset Reset for existing Consumer Group #11 41.40 note: This is an issue with the package mentioned above, not pip.
  Auto Offset Reset for existing Consumer Group #11 41.40 hint: See above for output from the failure.

[OneCricketeer]

Related #1190 #1182 #1466 #1439

[LiamClarkeNZ]

Yep, you'll need to download the librdkafka library, so that CKP can link against it when building from source. So apt-get/dnf/pacman whatever install librdkafka. If there are no existing packages for it that support ARM on your Docker image's OS, you can built it from source. This is what we do:

# Install librdkafka from source for confluent-kafka-python
RUN LIBRDKAFKA_VERSION=$( curl -sL https://api.github.com/repos/edenhill/librdkafka/releases/latest \
    | jq -r ".tag_name") \ 
    && mkdir /tmp/librdkafka \
    && curl -L https://github.com/edenhill/librdkafka/archive/refs/tags/$LIBRDKAFKA_VERSION.tar.gz \
          | tar -xz -C /tmp/librdkafka --strip-components=1 \
    && cd /tmp/librdkafka \
    && ./configure --install-deps \
    && make \
    && make install

You'll also need to ensure that the compiler can find them at library install time, if they're installed to a non-traditional location. E.g., when I was trying to install CKP locally for an older Python that there were no binary wheels for ARM on:

brew install librdkafka
# Set these env vars for GCC to find librdkafka headers
env C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.9.2/include \ 
      LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.9.2/lib \
      pip install confluent-kafka[avro,json,protobuf]==1.9.2

[sivankumar86]

have you tried this ?

FROM python:3.9.11-alpine3.15 

# upgrade pip to the latest version
RUN apk --no-cache upgrade \
    && pip install --upgrade pip \
    && apk --no-cache add tzdata build-base gcc libc-dev g++ make git bash

RUN git clone https://github.com/edenhill/librdkafka.git && cd librdkafka \
    && git checkout tags/v2.2.0 && ./configure --clean \
    && ./configure --prefix /usr/local \
    && make && make install
RUN pip install confluent-kafka==2.2.0