HiGHS fails to link with "undefined reference" errors

Question

On Arch Linux x86_64, Rust 1.66.0 stable, g++ 12.2.0.

I have a problem linking the good-lp crate, and narrowed it down to its indirect dependency highs-sys, even outside my own code.

This happens on a clean checkout of the latest master (c785240), as well as v1.2.2, as well as v0.2.1 (which I used with great success for months through good-lp). So it isn't a new problem in highs-sys, and might very well be a problem with my toolchain.

Versions:

$ cargo --version
cargo 1.66.0 (d65d197ad 2022-11-15)
$ rustc --version
rustc 1.66.0 (69f9c33d7 2022-12-12)
$ cc --version
cc (GCC) 12.2.0
...
$ ld --version
GNU ld (GNU Binutils) 2.39.0
...

Cloning and building:

$ git clone --recursive https://github.com/rust-or/highs-sys
$ cd highs-sys
$ cargo build
...
    Finished dev [unoptimized + debuginfo] target(s) in 3m 35s
$ cargo test --lib
    Finished test [unoptimized + debuginfo] target(s) in 0.04s
     Running unittests src/lib.rs (target/debug/deps/highs_sys-96b0b941fcc31775)

running 2 tests
test bindgen_test_layout___fsid_t ... ok
test bindgen_test_layout_imaxdiv_t ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

All fine so far. The problem happens when we try to use the generated library, even with its own integration tests:

$ cargo test --test test_highs_call
   Compiling highs-sys v1.2.2 (/tmp/highs-sys)
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "/tmp/rustccbaKBt/symbols.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.110gvlokoswtomfn.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.13y9v90q7e3dvfo2.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.17wo7aophf8in4vn.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.1cdkl7c2apbn7crn.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.1dj1sbwseam0n9ft.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.1hgigplg6qv30idz.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.1ms9q0du418ic26n.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.1tcgu3qw79kuuw1h.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.1vs3h8jyr2rojt51.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.20xdzk038hc30h4j.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.25ph4lcxn1xxn6fn.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.27am5w6h2agsxcl7.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2djhnnv2jm2ibcl3.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2i1j9l2lu5t0k0vu.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2meo5gq21h6h8bqe.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2ol9s57flfu167qp.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2s2g91x597h8e2ey.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2v1cfn4oct7skc3q.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2v9apdn2isgwdl5q.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2wpeqt6tzxghqi56.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2yk7p7w6ixraj427.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2z4b238c1zbjnij6.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.2zlr9mnhg9hyf02g.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.32cfoccwtdvhq06x.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.33nztruk7k2tts8x.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.3cvdzi91jork7mds.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.3fb9xf8nxxzcec41.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.3is6rgf4g5g16vl3.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.3s3dhxd64ny5tive.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.3wn5g0h8q5u3bxky.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4150s6y93i5ma35t.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4335j389sygn8m99.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.44i1r6id1s31acf1.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4injouqsnun7hcvt.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4nb66ozge3nmeuwq.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4p2li2uv9dxcrx2d.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4pz1m3mlzi8imskl.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4tg55omun1u77yh3.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4thto7zkdscqxses.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4vrsyb4og25lcxo8.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.4z0e0rsh1apo67mr.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.50cxv68n1h5055a5.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.515xy9o2bj3al2ry.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.59no2g8u5zmbleot.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.5b574862nrupsals.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.5fu18wwg3p5nmrsh.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.5gi5ablibidmtrpd.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.kjht10drvblu37w.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.ugtef95uy6fok7q.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.xcjfvort5vhydtl.rcgu.o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.3mrzemp57wngq71t.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/highs-sys/target/debug/deps" "-L" "/tmp/highs-sys/target/debug/build/highs-sys-21b14ce07e65289c/out/lib" "-L" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-5678a62cd2e949f2.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-cafb712826ee2a26.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunicode_width-bb4ac1585c9c1153.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_std-69fab17bbe87f87e.rlib" "/tmp/highs-sys/target/debug/deps/libhighs_sys-addd17046a23d2a8.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-a11e3ca400b3ed09.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-3e82a3fced649488.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-53a4330185981bcb.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-2a8b57667b4852b5.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-9370462deca12c5a.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-7da763b8d3620472.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-5bde27582a7f5af7.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-1204e05b2d47e3d7.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-43987de2766b6923.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-d6499a0705316aa5.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c9a27c90d8fbf11e.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-8f159929cbfdfaf1.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-d2f1e8f3bb5cba95.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-9862f486269f442f.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-0434381f2f012ae2.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-68549403a59fd02e.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-4cefb2045f924a5b.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-272615fc4f10c50d.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-860619b93700e7eb.rlib" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b73e5b4656934876.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lgomp" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/thomas/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs"
  = note: /usr/bin/ld: /tmp/highs-sys/target/debug/deps/test_highs_call-e669101ae4108d78.5gi5ablibidmtrpd.rcgu.o: in function `test_highs_call::highs_call':
          /tmp/highs-sys/tests/test_highs_call.rs:88: undefined reference to `Highs_lpCall'
          collect2: error: ld returned 1 exit status
          
  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `highs-sys` due to previous error

Similar output happens for the other integration test, test_highs_functions, but with undefined references to all Highs_* functions used in that file.

Verbose build output:

$ cargo build --verbose --test test_highs_call
...
   Compiling highs-sys v1.2.2 (/tmp/highs-sys)
     Running `rustc --crate-name test_highs_call --edition=2018 tests/test_highs_call.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test -C metadata=e669101ae4108d78 -C extra-filename=-e669101ae4108d78 --out-dir /tmp/highs-sys/target/debug/deps -C incremental=/tmp/highs-sys/target/debug/incremental -L dependency=/tmp/highs-sys/target/debug/deps --extern highs_sys=/tmp/highs-sys/target/debug/deps/libhighs_sys-addd17046a23d2a8.rlib -L native=/tmp/highs-sys/target/debug/build/highs-sys-21b14ce07e65289c/out/lib`
...
... and then the above error again 

That rustc command, when run manually, also fails with the same error.

Running the cc command manually fails due to /tmp/rustccbaKBt/symbols.o having been removed, but if I remove that from the command line, the same error occurs again.

The symbol is mentioned as undefined (U) in this file:

$ nm target/debug/deps/test_highs_call-e669101ae4108d78.5gi5ablibidmtrpd.rcgu.o | grep Highs_lpCall
                 U Highs_lpCall

And it gets defined (T) in this one, which occurs later on the command line, as it should:

$ nm /tmp/highs-sys/target/debug/deps/libhighs_sys-addd17046a23d2a8.rlib | grep Highs_lpCall
nm: lib.rmeta: no symbols
00000000 T Highs_lpCall

However, all the symbols appear at address 00000000 which seems suspicious to my uninformed eye.

I'm honestly not sure if this is a problem with highs, but other libraries that link to C APIs are working fine for me.

Also tried with gcc11, which is still available in the repositories as a separate package:

$ export CC=/usr/bin/gcc-11
$ export CXX=/usr/bin/g++-11
$ export RUSTFLAGS="-C linker=/usr/bin/gcc-11"

This makes it also error out with "undefined reference" errors. Which is weird, because I've been using GCC 11 for quite a while.

But when I set these variables to use clang instead of gcc...

$ export CC=/usr/bin/clang
$ export CXX=/usr/bin/clang++
$ export RUSTFLAGS="-C link-arg=-fuse-ld=lld"

... then it links successfully and tests are green! But this isn't a proper solution.

Any idea what more I could try?

Answer 1

It turns out this was due to link-time optimization (LTO).

libhighs.a was compiled from C++ code by GCC using -flto -fno-fat-lto-objects, causing it to emit only GCC intermediate bytecode into the archive, but not actual machine code. These flags were added by CMake through its option INTERPROCEDURAL_OPTIMIZATION.

But because LTO was not enabled during linking, no machine code was being produced at link time either. Normally the GNU linker would take care of that, but I think the bytecode might have gotten unusable or lost when libhighs.a was linked into a .rlib crate (hypothesis, didn't check). And I don't want to use the GNU linker for Rust code anyway because then I would lose the option to use LTO on Rust code (which always uses LLVM bitcode).

So for now, I think the best solution is to disable LTO on the C++ HiGHS build. I filed an issue with highs-sys and chimed in on an issue with HiGHS to make LTO more optional.