glibc vs. musl (shared) binary compatibility

Question

In the section "Is musl compatible with glibc?" the musl FAQ states that:

Binary compatibility is much more limited, but it will steadily increase with new versions of musl. At present, some glibc-linked shared libraries can be loaded with musl, but all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.

How can I make this more tractable in terms of functions and definitions that I can use, and compiler or linker options that I need to set to get this compatibility.

What is an example of "all but the simplest" applications? Is there actually some minimal (possibly only academic) scenario that I can construct, where I build a C binary that has shared linkage against libc and which works both with glibc and musl?

Or in other words, can I construct some minimal C binary that uses at least one function from libc, is not statically linked, and that I can drop into both a Debian and Alpine environment and it will be able to start?

This is mainly an academic exercise to help me understand the differences between both libraries.

Somehow I am having a hard time accepting, that it should be easier to create a portable POSIX shell script than a portable C application.

Answer 1

Or in other words, can I construct some minimal C binary that uses at least one function from libc, is not statically linked, and that I can drop into both a Debian and Alpine environment and it will be able to start?

Not a C binary. Your binary just can't use anything from GLIBC, and a C binary will use at least crt0.o.

You can however construct such binary in assembly. E.g. a binary built from this source and using nasm

    SECTION .text
    global _start
_start:
    mov ebx,0       ; exit code
    mov eax,1       ; SYS_exit
    int 0x80        ; syscall

wouldn't care whether it's linked statically or dynamically.

On the other hand, a binary built from this source:

.text
    .align 4
    .globl main
main:
     pushl %ebp
     movl %esp,%ebp
     xorl %eax,%eax
     leave
     ret

is not guaranteed to work.

What's the difference? The first binary will not have crt0.o linked into it, and should work fine when linked dynamically so long as libc.so itself initializes correctly.

Answer 2

I played a bit more and got a binary that works on my Gentoo machine and an Alpine docker container. It required (in Alpine) creating symlinks to the dynamic loader and libc using the names expected by my Gentoo system (/lib64/ld-linux-x86-64.so.2 and /lib64/libc.so.6), but my test program worked as expected:

#include <stdio.h>

int main() {
    printf("Hello, world\n");
    return 0;
}

I can probably get a program compiled in Alpine to work on Gentoo as well, but I haven't played with the symlinks yet. I also tested a program that claims to implement producers consumers, and that appears to work as well.

As for the limits of what will work, I suspect that's going to depend a lot on each libc. If glibc changes pthread_create to be a macro directly calling something else, I think that's legal but that would cause issues with musl binary compatibility.

I'm not sure you can do this with any generic enviornment and dynamic linkage, even with clever use of LD_PRELOAD (didn't test this too heavily though). And as a practical matter, I suspect far fewer problems just compling one version against glibc, and another against musl.