Why does lxml compilation look for 32-bit libraries?

Question

The installation fails saying 32-bit .so file is not compatible:

gcc -pthread -shared -Wl,-z,relro -Wl,-z,now -g -Wl,-z,relro -Wl,-z,now -g build/temp.linux-x86_64-3.8/src/lxml/etree.o -L/usr/lib64 -L/usr/lib64 -lxslt -lexslt -lxml2 -lrt -lz -lm -o build/lib.linux-x86_64-3.8/lxml/etree.cpython-38-x86_64-linux-gnu.so
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/8/../../../libxml2.so when searching for -lxml2
/usr/bin/ld: skipping incompatible //lib/libxml2.so when searching for -lxml2
/usr/bin/ld: skipping incompatible //usr/lib/libxml2.so when searching for -lxml2
/usr/bin/ld: cannot find -lxml2
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

Why is it searching in /usr/lib? Shouldn't 64-bit images look for libraries in /usr/lib64 unless set otherwise (even gcc says to look there)? I tried setting LD_LIBRARY_PATH but that didn't help.

This is the Dockerfile I use.

FROM registry.access.redhat.com/ubi8/ubi-minimal

RUN microdnf -y install python38

RUN microdnf -y install shadow-utils

RUN microdnf -y install --nodocs \
    gcc-c++                      \
    python38-devel               \
    python38-wheel               \
    unixODBC-devel               \ 
    bzip2                        \
    bzip2-devel                  \
    expat                        \
    expat-devel                  \
    gcc                          \
    git                          \
    glibc-langpack-en            \
    libffi                       \
    libffi-devel                 \
    libxml2                      \
    libxml2-devel                \
    libxslt                      \
    libxslt-devel                \
    unzip                        \
    wget                         \
    yum-utils                    \
    zip                          \
    make                         \
    openssl                      \
    openssl-devel                \
    sqlite-devel        

RUN pip3 install --no-binary :all: lxml==4.6.2

Answer 1

1. Disable libxslt's gcrypt functionality by changing one line in buildlibxml.py (next to setup.py):

   if sys.platform in ('darwin',):" to just "if True:

This adds --without-crypto to libxslt's configure command, which is necessary because it will try to link in 64-bit libraries that won't play well with the 32-bit static build we're doing

2. Run this command:

   LDFLAGS='-m32 -Wl,--no-as-needed' CFLAGS='-m32 -lrt' LIBRARY="./build/tmp/libxml2/lib" python setup.py build --static-deps --static --libxml2-version=2.9.2 --libxslt-version=1.1.28 --libiconv-version=1.14 --without-cython

If you encounter problems, replace the library versions with the ones you find in ./libs.

• Without --without-cython, if the cython command is found, setup.py will try to use it, regardless of what version of Python the cython was built with/for.

• Without LIBRARY=... setup.py will complain that it's not configured for a static build.

• Without the -m32s the dependencies will get built for 64-bit and your linker will skip over them.

• Without -lrt and the -Wl,--no-as-needed you'll get ImportError: lxml/etree.so: undefined symbol: clock_gettime

• Maybe this works without explicit version arguments

• This probably works without --static-deps and --static, but explicit is better than implicit and it doesn't do any harm

Celebrate:

$ python
Python 2.7.3 (default, Apr  2 2014, 18:00:22) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml.etree
>>>

(It imported, so it worked!)

Answer 2

The problem seems to be happening because microdnf installs libxml2 and libxml2-devel 32-bit in the initial installation:

[root@364c2579ba54 /]# rpm -qa | grep libxml
libxml2-2.9.7-15.el8_7.1.x86_64
libxml2-2.9.7-16.el8.i686
libxml2-devel-2.9.7-16.el8.i686

It is possible that this situation arises due to the presence of a 64-bit version of libxml2 in the image. Therefore, when executing the command microdnf install libxml2 libxml2-devel it assumes that you (now) require the installation of 32-bit version. This behavior differs from CentOS. If you use microdnf install libxml2-devel, it will install the 64-bit version.