LD_PRELOAD causing segmentation fault in dynamic library loader

Question

I have written a library which is intended to be loaded via LD_PRELOAD. On some Linux systems, this is causing the dynamic library loader to segfault during initialisation.

I have a simple test case that exhibits this behaviour, but only if I link with -lm. For example:

# Works fine
gcc -o vecadd.normal -std=c99 vecadd.c -lOpenCL
LD_PRELOAD=/path/to/my/library.so ./vecadd.normal

# Causes segmentation fault
gcc -o vecadd.broken -std=c99 vecadd.c -lOpenCL -lm
LD_PRELOAD=/path/to/my/library.so ./vecadd.broken

The strange thing about this is that libm.so seems to be included in both versions: ldd shows exactly the same set of libraries, just in a different order:

vecadd.normal:
  linux-vdso.so.1 =>  (0x00007fffed9ff000)
  libOpenCL.so => /usr/lib64/libOpenCL.so (0x00007f135c9b1000)
  libc.so.6 => /lib64/libc.so.6 (0x00007f135c61c000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007f135c418000)
  libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f135c20f000)
  libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f135bf08000)
  libm.so.6 => /lib64/libm.so.6 (0x00007f135bc84000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f135ba6e000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f135cbd4000)
vecadd.broken:
  linux-vdso.so.1 =>  (0x00007fff25c74000)
  libOpenCL.so => /usr/lib64/libOpenCL.so (0x00007fb8c071e000)
  libm.so.6 => /lib64/libm.so.6 (0x00007fb8c0499000)
  libc.so.6 => /lib64/libc.so.6 (0x00007fb8c0105000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007fb8bff01000)
  libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007fb8bfcf7000)
  libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fb8bf9f1000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb8bf7db000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fb8c0941000)

Google directed me to LD_DEBUG, which also points to libm.so as part of the problem:

 14143: symbol=fma;  lookup in file=./vecadd.broken [0]
 14143: symbol=fma;  lookup in file=/path/to/my/library.so [0]
 14143: symbol=fma;  lookup in file=/usr/lib64/libOpenCL.so [0]
 14143: symbol=fma;  lookup in file=/lib64/libm.so.6 [0]
 14143: binding file /path/to/my/library.so [0] to /lib64/libm.so.6 [0]: normal symbol `fma' [GLIBC_2.2.5]
Segmentation fault (core dumped)

Unfortunately the only machines that I can reproduce this problem on don't appear to have debug symbols for the dynamic library loader available (and I don't have any administrative permissions), so GDB doesn't yield anything useful:

gdb ./vecadd.broken
(gdb) set environment LD_PRELOAD /path/to/my/library.so
(gdb) run
Starting program: vecadd.broken 

Program received signal SIGSEGV, Segmentation fault.
0x0000000000003dce in ?? ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.1.x86_64
(gdb) bt
#0  0x0000000000003dce in ?? ()
#1  0x00007fff7f755369 in ?? ()
#2  0x00007fffffffd840 in ?? ()
#3  0x00007fff7fde9e91 in _dl_relocate_object ()
   from /lib64/ld-linux-x86-64.so.2
#4  0x00007fff7fde18a3 in dl_main () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fff7fdf3a0e in _dl_sysdep_start () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fff7fddf4a4 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fff7fddeb08 in _start () from /lib64/ld-linux-x86-64.so.2
#8  0x0000000000000001 in ?? ()
#9  0x00007fffffffde01 in ?? ()
#10 0x0000000000000000 in ?? ()

What's the best way to go about finding the root cause of this problem?

Answer 1

You can upload your own dynamic loader built with symbols and call it to link your app, like:

∞ /lib/ld-linux-x86-64.so.2 /bin/echo linked linked