Dynamic linking in Python CFFI?

Question

I'm trying to use CFFI for dynamic linking of dependent C libraries in Python; am I mis-understanding it?

In the following extremely simplified example, library foo_b depends on library foo_a. Specifically, it depends on bar, and it exposes its own function baz.

from cffi import FFI
from pathlib import Path

Path('foo_a.h').write_text("""\
int bar(int x);
""")
Path('foo_a.c').write_text("""\
#include "foo_a.h"
int bar(int x) {
  return x + 69;
}
""")

Path('foo_b.h').write_text("""\
int baz(int x);
""")
Path('foo_b.c').write_text("""\
#include "foo_a.h"
#include "foo_b.h"
int baz(int x) {
  return bar(x * 100);
}
""")

ffi_a = FFI()
ffi_b = FFI()

ffi_a.cdef('int bar(int x);')
ffi_a.set_source('ffi_foo_a', '#include "foo_a.h"', sources=['foo_a.c'])
ffi_a.compile()

ffi_b.cdef('int baz(int x);')
ffi_b.include(ffi_a)
ffi_b.set_source('ffi_foo_b', '#include "foo_b.h"', sources=['foo_b.c'])
ffi_b.compile()

import ffi_foo_a
if ffi_foo_a.lib.bar(1) == 70: print('foo_a OK')
else: raise AssertionError('foo_a ERR')

import ffi_foo_b  # Crashes on _this_ line due to undefined symbol "bar", DESPITE the fact that we included ffi_a, which should provide that symbol
if ffi_foo_b.lib.baz(420) == 42069: print('foo_b OK')
else: raise AssertionError('foo_b ERR')

However, it doesn't compile, instead crashing on the indicated line with the indicated error message.

I don't understand why this example isn't working, considering the following in the CFFI documentation:

For out-of-line modules, the ffibuilder.include(other_ffibuilder) line should occur in the build script, and the other_ffibuilder argument should be another FFI instance that comes from another build script. When the two build scripts are turned into generated files, say _ffi.so and _other_ffi.so, then importing _ffi.so will internally cause _other_ffi.so to be imported. At that point, the real declarations from _other_ffi.so are combined with the real declarations from _ffi.so.

If ffibuilder.include() isn't the right way to dynamically link together multiple CFFI-based libraries, what is?

Or if ffibuilder.include() is the right way to dynamically link together multiple CFFI-based libraries, what am I doing wrong?

Answer 1

The library ffi_foo_b.cpython-XXX.so fails to import because of the C-level issue that it doesn't find the symbol bar. Looking at the tests in CFFI, it seems that this case is not supported: the ffi_foo_b.include(ffi_foo_a) syntax is not enough to cause the C-level ffi_foo_b.cpython-XXX.so to be compiled specially. If a symbol from ffi_foo_a.cpython-XXX.so is needed at the C level for ffi_foo_b.cpython-XXX.so to load, then it won't work. The CFFI documentation is kinda misleading. It means rather that, say, you can take things like struct type definitions that appear in ffi_a and use them from lib_foo_b.ffi.

In terms of CFFI implementation, I'm not quite sure how this case could be supported: for example, on Windows you need special tricks to export a symbol from a DLL (with the extension .dll or .pyd for CPython extension modules). In other words your example on Windows produces a ffi_foo_a that doesn't automatically export bar at all at the level of C. You can still call ffi_foo_a.bar(), because the look up of bar inside ffi_foo_a is done at a different level (on any platform) than the raw C level. You could also call it as ffi_foo_b.bar() thanks to the ffi.include(), if there wasn't the problem that ffi_foo_b tries to use bar at the C level directly. But you can't use the C symbol bar from foo_b.set_source().

For now I'd recommend one of these solutions:

1. consolidating everything inside a single ffi.

2. make standard, CFFI- and Python-independant .so as you need them, e.g. foo_a.so and foo_b.so, with the proper C-level Makefile or something, with foo_b compiled in such a way to say that it depends on foo_a---using some gcc arguments like -L . -lfoo_a.so. Then you can wrap these two .so with two CFFI libraries. You can use ffi.include() then; this is really how ffi.include() was meant to work.

3. as a mixture of 1 and 2, you can probably have foo_a.so compiled as a standalone module, and then wrapped in ffi_a, but keep the existing solution for ffi_b because no symbol from ffi_b is expected to be found at the C level from somewhere else.

EDIT: two more possible solutions:

4. You can probably have it working just like you wrote, but you need to add platform- and compiler-specific options. I'd not recommend that option.

5. You can bypass the problem by not calling bar() from foo_b.c. Instead, you'd write code like this:

    Path('foo_b.c').write_text("""\
    #include "foo_b.h"
    static int (*_glob_bar)(int);  // global variable
    int baz(int x) {
      return _glob_bar(x * 100);
    }
    """)
   
    ffi_b.cdef("""
        int (*_glob_bar)(int);
        int baz(int x);
    """)
   

   This removes the dependency at the C level. You need to initialize the global variable once after import:

    import ffi_foo_b
    ffi_foo_b.lib._glob_bar =
        ffi_foo_a.ffi.addressof(ffi_foo_a.lib, "bar")