TechQA.

Confluent-kafka multistage build issue

63 views Asked by At

I have this multistage docker build as follows,

# Use an Amazon Linux 2 base image
FROM amazonlinux:2.0.20230418.0 AS builder

# Install build tools and dependencies
RUN yum install -y gcc-c++ make openssl-devel zlib-devel librdkafka-devel && yum groupinstall -y  "Development Tools"

# Download librdkafka 2.2.0 source
RUN cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0 \
    && ./configure --prefix /usr && make -j 3 && make install && ldconfig

# Build librdkafka
WORKDIR /usr/src/librdkafka-2.2.0
RUN ./configure --prefix=/usr \
    && make \
    && make install

# Final image
FROM amazonlinux:2.0.20230418.0

# Copy only librdkafka binaries from builder stage
COPY --from=builder /usr/lib/librdkafka* /usr/lib/
COPY --from=builder /usr/include/librdkafka /usr/include/librdkafka

# Set LD_LIBRARY_PATH to include librdkafka library
ENV LD_LIBRARY_PATH=/usr/lib:$LD_LIBRARY_PATH

# Install Python and required packages
RUN yum install -y python2.7 python-pip

# Install confluent-kafka Python library
RUN pip install confluent-kafka=2.2.0

# Copy your application code
COPY . /app
WORKDIR /app

# Command to run your application
CMD ["python", "main.py"]

where i'm trying to install confluent-kafka after building librdkafka from source for some reson i'm always ended up with the following error

Agent pid 2036891
Identity added: /home/john.doe/.ssh/id_rsa (john.doe@abc-corp)
#1 [internal] load build definition from Dockerfile
#1 sha256:d2adfb92a6221bec17bf4c12942bbff41d7b7280b2cf1b54bcb01e340ea55d1d
#1 transferring dockerfile: 3.27kB done
#1 DONE 0.0s
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#REMVOED THESE LOGS AS IT WAS A SUCCESS
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# 
#
#11 84.51 Complete!
#11 DONE 85.3s

#12 [builder 3/5] RUN cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig

#12 37.40 Turn off this advice by setting config variable advice.detachedHead to false
#12 37.40 
#12 37.40 HEAD is now at e75de5be Generates a random salt only when (#4350)
#12 38.27 checking for OS or distribution... ok (amzn)
#12 38.27 checking for C compiler from CC env... failed
#12 38.28 checking for gcc (by command)... ok
#12 38.29 checking for C++ compiler from CXX env... failed
#12 38.30 checking for C++ compiler (g++)... ok
#12 38.32 checking executable ld... ok
#12 38.33 checking executable nm... ok
#12 38.34 checking executable objdump... ok
#12 38.35 checking executable strip... ok
#12 38.36 checking executable libtool... ok
#12 38.36 checking executable ranlib... ok
#12 38.37 checking for pkgconfig (by command)... ok
#12 38.38 checking for install (by command)...mklove/modules/configure.base: line 1181: which: command not found
#12 38.38  failed
#
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# REMOVED A BUNCH OF LOGS HERE
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#
-Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c MetadataImpl.cpp -o MetadataImpl.o
#12 63.97 Generating pkg-config file rdkafka++.pc
#12 63.97 Generating pkg-config file rdkafka++-static.pc
#12 64.36 Creating shared library librdkafka++.so.1
#12 64.36 Creating static library librdkafka++.a
#12 64.36 g++  -shared -Wl,-soname,librdkafka++.so.1 RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o -o librdkafka++.so.1 -L../src -lrdkafka
#12 64.36 ar rcs librdkafka++.a RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o
#12 64.38 cp librdkafka++.a librdkafka++-dbg.a
#12 64.46 cp librdkafka++.so.1 librdkafka++-dbg.so.1
#12 64.47 Creating librdkafka++.so symlink
#12 64.47 rm -f "librdkafka++.so" && ln -s "librdkafka++.so.1" "librdkafka++.so"
#12 64.47 Checking librdkafka++ integrity
#12 64.47 librdkafka++.so.1              OK
#12 64.47 librdkafka++.a                 OK
#12 64.47 make[1]: Leaving directory `/opt/librdkafka/src-cpp'
#12 64.47 make -C examples
#12 64.48 make[1]: Entering directory `/opt/librdkafka/examples'
#12 64.48 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_example.c -o rdkafka_example  \
#12 64.48       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 64.48 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_performance.c -o rdkafka_performance  \
#12 64.48       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 64.48 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_example.cpp -o rdkafka_example_cpp  \
#12 64.48       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 64.88 # rdkafka_example is ready
#12 64.89 #
#12 64.89 # Run producer (write messages on stdin)
#12 64.89 ./rdkafka_example -P -t <topic> -p <partition>
#12 64.89 
#12 64.89 # or consumer
#12 64.89 ./rdkafka_example -C -t <topic> -p <partition>
#12 64.89 
#12 64.89 #
#12 64.90 # More usage options:
#12 64.90 ./rdkafka_example -h
#12 64.90 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_complex_consumer_example.c -o rdkafka_complex_consumer_example  \
#12 64.90       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 65.08 # rdkafka_performance is ready
#12 65.08 #
#12 65.08 # Run producer
#12 65.08 ./rdkafka_performance -P -t <topic> -p <partition> -s <msgsize>
#12 65.08 
#12 65.08 # or consumer
#12 65.08 ./rdkafka_performance -C -t <topic> -p <partition>
#12 65.08 
#12 65.09 #
#12 65.09 # More usage options:
#12 65.09 ./rdkafka_performance -h
#12 65.09 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_complex_consumer_example.cpp -o rdkafka_complex_consumer_example_cpp  \
#12 65.09       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 65.30 # rdkafka_complex_consumer_example is ready
#12 65.31 #
#12 65.31 ./rdkafka_complex_consumer_example <topic[:part]> <topic2[:part]> ..
#12 65.31 
#12 65.31 #
#12 65.31 # More usage options:
#12 65.31 ./rdkafka_complex_consumer_example -h
#12 65.31 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp kafkatest_verifiable_client.cpp -o kafkatest_verifiable_client  \
#12 65.31       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 65.65 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src producer.c -o producer  \
#12 65.65       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 65.95 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src consumer.c -o consumer  \
#12 65.95       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 66.14 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src idempotent_producer.c -o idempotent_producer  \
#12 66.14       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 66.23 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src transactions.c -o transactions  \
#12 66.23       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 66.41 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src delete_records.c -o delete_records  \
#12 66.41       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 66.57 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp openssl_engine_example.cpp -o openssl_engine_example_cpp  \
#12 66.57       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 66.71 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src list_consumer_groups.c -o list_consumer_groups  \
#12 66.71       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 67.15 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src describe_consumer_groups.c -o describe_consumer_groups  \
#12 67.15       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 67.41 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src list_consumer_group_offsets.c -o list_consumer_group_offsets  \
#12 67.41       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 67.51 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src alter_consumer_group_offsets.c -o alter_consumer_group_offsets  \
#12 67.51       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 67.78 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src incremental_alter_configs.c -o incremental_alter_configs  \
#12 67.78       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 67.80 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src user_scram.c -o user_scram  \
#12 67.80       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 67.91 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src misc.c -o misc  \
#12 67.91       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
#12 68.19 make[1]: Leaving directory `/opt/librdkafka/examples'
#12 68.20 Updating CONFIGURATION.md
#12 68.25 Checking  integrity
#12 68.25 CONFIGURATION.md               OK
#12 68.25 examples/rdkafka_example       OK
#12 68.25 examples/rdkafka_performance   OK
#12 68.25 examples/rdkafka_example_cpp   OK
#12 68.46 make[1]: Entering directory `/opt/librdkafka/src'
#12 68.46 Checking librdkafka integrity
#12 68.47 librdkafka.so.1                OK
#12 68.47 librdkafka.a                   OK
#12 68.47 Symbol visibility              OK
#12 68.47 make[1]: Leaving directory `/opt/librdkafka/src'
#12 68.49 make[1]: Entering directory `/opt/librdkafka/src-cpp'
#12 68.50 Generating pkg-config file rdkafka++-static.pc
#12 68.50 Checking librdkafka++ integrity
#12 68.50 librdkafka++.so.1              OK
#12 68.50 librdkafka++.a                 OK
#12 68.50 make[1]: Leaving directory `/opt/librdkafka/src-cpp'
#12 68.54 make[1]: Entering directory `/opt/librdkafka/src'
#12 68.54 Install librdkafka to /usr
#12 68.54 d $DESTDIR/usr/include/librdkafka
#12 68.54 /bin/sh: d: command not found
#12 68.54 make[1]: [lib-install] Error 127 (ignored)
#12 68.54 d $DESTDIR/usr/lib
#12 68.54 /bin/sh: d: command not found
#12 68.54 make[1]: [lib-install] Error 127 (ignored)
#12 68.54 rdkafka.h rdkafka_mock.h $DESTDIR/usr/include/librdkafka
#12 68.54 /bin/sh: rdkafka.h: command not found
#12 68.54 make[1]: *** [lib-install] Error 127
#12 68.54 make[1]: Leaving directory `/opt/librdkafka/src'
#12 68.54 make: *** [install-subdirs] Error 2
#12 ERROR: executor failed running [/bin/sh -c cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig]: exit code: 2
------
 > [builder 3/5] RUN cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig:
------
executor failed running [/bin/sh -c cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig]: exit code: 2


kafka_test                                           Error on build

job_done

any help would be appreciated, thanks

docker cmake docker-multi-stage-build confluent-kafka-python librdkafka
Original Q&A
1

There are 1 answers

2
OneCricketeer On

Your error is here

#12 68.54 /bin/sh: d: command not found
#12 68.54 make[1]: [lib-install] Error 127 (ignored)
#12 68.54 d $DESTDIR/usr/lib
#12 68.54 /bin/sh: d: command not found
#12 68.54 make[1]: [lib-install] Error 127 (ignored)

That should be cd command, not d...

You should install from YUM repos, not clone and build from source.

Otherwise, confluent-kafka-python automatically includes it with binary wheels

Related Questions in DOCKER

Related Questions in CMAKE

Related Questions in DOCKER-MULTI-STAGE-BUILD

Related Questions in CONFLUENT-KAFKA-PYTHON

Related Questions in LIBRDKAFKA

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions