ldd shows .so not found but RUNPATH contains the lib

Question

I'm trying to get some binaries running on NixOS, and I ran into a weird situation when trying to get ldd to find libpython2.7.so.1.0 and other libs.

$ ldd lldb
./lldb: /nix/store/5rjfisjzz6vgwmgf7zx25yd9p6rfs0zy-ncurses-6.2-abi5-compat/lib/libtinfo.so.5: no version information available (required by ./lldb)
./lldb: /nix/store/5rjfisjzz6vgwmgf7zx25yd9p6rfs0zy-ncurses-6.2-abi5-compat/lib/libtinfo.so.5: no version information available (required by /nix/store/nn8pr7xzam0rz7fq95x9dpi087xazsnv-theos/share/theos/toolchain/linux/iphone/usr/bin/./../lib/liblldb.so.10git)
    linux-vdso.so.1 (0x00007ffd57b80000)
    libpthread.so.0 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libpthread.so.0 (0x00007f921e821000)
    liblldb.so.10git => /nix/store/nn8pr7xzam0rz7fq95x9dpi087xazsnv-theos/share/theos/toolchain/linux/iphone/usr/bin/./../lib/liblldb.so.10git (0x00007f9215c3c000)
    libz.so.1 => /nix/store/1l4r0r4ab3v3a3ppir4jwiah3icalk9d-zlib-1.2.11/lib/libz.so.1 (0x00007f9215c1f000)
    librt.so.1 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/librt.so.1 (0x00007f9215c12000)
    libdl.so.2 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libdl.so.2 (0x00007f9215c0d000)
    libtinfo.so.5 => /nix/store/5rjfisjzz6vgwmgf7zx25yd9p6rfs0zy-ncurses-6.2-abi5-compat/lib/libtinfo.so.5 (0x00007f9215ba7000)
    libm.so.6 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libm.so.6 (0x00007f9215a66000)
    libstdc++.so.6 => /nix/store/c9f15p1kwm0mw5p13wsnvd1ixrhbhb12-gcc-10.3.0-lib/lib/libstdc++.so.6 (0x00007f9215891000)
    libgcc_s.so.1 => /nix/store/c9f15p1kwm0mw5p13wsnvd1ixrhbhb12-gcc-10.3.0-lib/lib/libgcc_s.so.1 (0x00007f9215875000)
    libc.so.6 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libc.so.6 (0x00007f92156b0000)
    /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/ld-linux-x86-64.so.2 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib64/ld-linux-x86-64.so.2 (0x00007f921e843000)
    libpython2.7.so.1.0 => not found
    libncurses.so.5 => not found
    libform.so.5 => not found
    libpanel.so.5 => not found
    libuuid.so.1 => not found
    libedit.so.2 => not found
    libxml2.so.2 => not found

There are quite a few missing, but let's focus on libpython2.7.so.1.0. When we check the RUNPATH of ld64,

$ readelf -d ld64 | rg RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/c9f15p1kwm0mw5p13wsnvd1ixrhbhb12-gcc-10.3.0-lib/lib:/nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib:/nix/store/xvyzi7cr0icnyavi5pm9rywjc4d8l7sx-libedit-20210714-3.1/lib:/nix/store/yxflij8cg4fgnzqmda91jx4d94jvkjf5-util-linux-2.37.2-lib/lib:/nix/store/370lxynzkmwrk8685jx9p2vgh7h0xp2h-libxml2-2.9.12/lib:/nix/store/5rjfisjzz6vgwmgf7zx25yd9p6rfs0zy-ncurses-6.2-abi5-compat/lib:/nix/store/nvx0l614cv661i5zz6w3j3y2w1xzppv1-python-2.7.18/lib:/nix/store/7344a20iqaja6i2qdz2xrgzy28rgnz5p-util-linux-2.37.2-lib/lib:/nix/store/1l4r0r4ab3v3a3ppir4jwiah3icalk9d-zlib-1.2.11/lib:$ORIGIN/../lib]

it contains /nix/store/nvx0l614cv661i5zz6w3j3y2w1xzppv1-python-2.7.18/lib, which has our desired lib,

$ file /nix/store/nvx0l614cv661i5zz6w3j3y2w1xzppv1-python-2.7.18/lib/libpython2.7.so.1.0
/nix/store/nvx0l614cv661i5zz6w3j3y2w1xzppv1-python-2.7.18/lib/libpython2.7.so.1.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

I'm not sure how this is possible. The executable is also 64-bit, as we can see here:

$ file lldb
lldb: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped

Answer 1

I'm not sure how this is possible.

There is a big difference between RUNPATH and RPATH: the former applies only to the binary itself (in this case, only to the lldb binary), the latter applies to the binary and all its dependencies.

Effectively using RUNPATH forces all binaries participating in the (dynamic) link to be linked correctly (to specify their own RUNPATH so that every one of their (recursive) dependencies is found).

What you observe is possible if:

1. lldb does not itself depend on libpython.2.7.so.1.0 and
2. one of lldbs dependencies does depend on on libpython.2.7.so.1.0 and
3. that dependency itself does not have a RUNPATH in which libpython.2.7.so.1.0 can be found.

You can verify that 1) is true with readelf -d ./lldb | grep libpython (I predict there will be no output).

If 1) is true, we know that 2 is true because libpython shows up in ldd output. You can find which of lldbs direct dependencies this is by going through them one by one (you can see them in the output from readelf -d lldb | grep NEEDED).

Once you found a dependency which needs libpython, you can confirm 3) by using readelf -d $direct_dependency | grep R.*PATH.