"sh: /usr/local/bin/node: not found" while it is present on the Filesystem

Question

After switching from bullseye-slim to bookworm-slim Docker image I experience env: can't execute 'node': No such file or directory error and it looks like it is caused by COPY --from=1234567890.dkr.ecr.us-west-2.amazonaws.com/sonarqube:latest / /.

Here is what I see on the container:

~ # ls -l /usr/local/bin/node
-rwxr-xr-x    1 root     root      92350992 Oct 13 14:02 /usr/local/bin/node
~ # /usr/local/bin/node
sh: /usr/local/bin/node: not found

The file is there, but cannot be executed as it is not found

My Dockerfile:

FROM node:18-bookworm-slim as base

WORKDIR /usr/src
EXPOSE 8080

COPY .npmrc /usr/src/
COPY package*.json /usr/src/

# --------------> The develop image
FROM base as develop

ENV NODE_ENV=development

RUN npm ci
COPY . /usr/src/

# --------------
FROM develop as develop_sonarqube

COPY --from=1234567890.dkr.ecr.us-west-2.amazonaws.com/sonarqube:latest / /

# --------------
FROM base as dependencies
RUN ...

If I build it targeting develop_sonarqube stage (this is where copy is), I get sh: /usr/local/bin/node: not found error when invoked with docker run --rm foo:baz npm run format

But if I target develop and run it, there is no error:

> [email protected] format
> prettier --check .

Checking formatting...
[warn] package-lock.json
[warn] package.json
[warn] Code style issues found in 2 files. Run Prettier to fix.

So it seams like COPY --from=1234567890.dkr.ecr.us-west-2.amazonaws.com/sonarqube:latest / / is a culprit, but it happens on bookworm-slim base image only meaning that if I change it back to bullseye-slim, it will work.

Any ideas how to fix it?

Debug info:

# ldd /usr/local/bin/node
    /lib64/ld-linux-x86-64.so.2 (0x7f000494b000)
    libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f000494b000)
Error loading shared library libstdc++.so.6: No such file or directory (needed by /usr/local/bin/node)
    libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f000494b000)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by /usr/local/bin/node)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f000494b000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f000494b000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /usr/local/bin/node)
Error relocating /usr/local/bin/node: _ZSt7getlineIcSt11char_traitsIcESaIcEERSt13basic_istreamIT_T0_ES7_RNSt7__cxx1112basic_stringIS4_S5_T1_EES4_: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE9pbackfailEi: symbol not found
Error relocating /usr/local/bin/node: _ZSt24__throw_out_of_range_fmtPKcz: symbol not found
Error relocating /usr/local/bin/node: _ZSt19__throw_logic_errorPKc: symbol not found
Error relocating /usr/local/bin/node: __once_proxy: symbol not found
Error relocating /usr/local/bin/node: _ZSt17__throw_bad_allocv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE6xsgetnEPcl: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertIyEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZdlPvm: symbol not found
Error relocating /usr/local/bin/node: _ZNSt6locale7classicEv: symbol not found
Error relocating /usr/local/bin/node: _Znwm: symbol not found
Error relocating /usr/local/bin/node: fcntl64: symbol not found
Error relocating /usr/local/bin/node: _ZNSt10_Sp_lockerC1EPKv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE5imbueERKSt6locale: symbol not found
Error relocating /usr/local/bin/node: _ZNSo5tellpEv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt13basic_filebufIcSt11char_traitsIcEE5closeEv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8__detail15_List_node_base7_M_hookEPS0_: symbol not found
Error relocating /usr/local/bin/node: gnu_get_libc_version: symbol not found
Error relocating /usr/local/bin/node: __cxa_demangle: symbol not found
Error relocating /usr/local/bin/node: _ZNSt9basic_iosIcSt11char_traitsIcEE7copyfmtERKS2_: symbol not found
Error relocating /usr/local/bin/node: _ZNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEE7_M_syncEPcmm: symbol not found
Error relocating /usr/local/bin/node: _ZNSolsEi: symbol not found
Error relocating /usr/local/bin/node: _ZSt16__throw_bad_castv: symbol not found
Error relocating /usr/local/bin/node: __cxa_guard_acquire: symbol not found
Error relocating /usr/local/bin/node: _ZNSt9basic_iosIcSt11char_traitsIcEE4initEPSt15basic_streambufIcS1_E: symbol not found
Error relocating /usr/local/bin/node: _ZdaPv: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertIlEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEED1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNKSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEE3strEv: symbol not found
Error relocating /usr/local/bin/node: backtrace_symbols: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertIbEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZSt20__throw_system_errori: symbol not found
Error relocating /usr/local/bin/node: _ZSt9use_facetISt5ctypeIcEERKT_RKSt6locale: symbol not found
Error relocating /usr/local/bin/node: _ZStrsIcSt11char_traitsIcESaIcEERSt13basic_istreamIT_T0_ES7_RNSt7__cxx1112basic_stringIS4_S5_T1_EE: symbol not found
Error relocating /usr/local/bin/node: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1ESt13_Ios_Openmode: symbol not found
Error relocating /usr/local/bin/node: _Znam: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertIPKvEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZSt11_Hash_bytesPKvmm: symbol not found
Error relocating /usr/local/bin/node: _ZNKSt5ctypeIcE13_M_widen_initEv: symbol not found
Error relocating /usr/local/bin/node: _ZSt18_Rb_tree_decrementPSt18_Rb_tree_node_base: symbol not found
Error relocating /usr/local/bin/node: _ZNSo5writeEPKcl: symbol not found
Error relocating /usr/local/bin/node: _ZNSo5flushEv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8__detail15_List_node_base11_M_transferEPS0_S1_: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertImEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZNSolsEs: symbol not found
Error relocating /usr/local/bin/node: _ZNSt12__basic_fileIcED1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt9basic_iosIcSt11char_traitsIcEE5imbueERKSt6locale: symbol not found
Error relocating /usr/local/bin/node: _ZdlPv: symbol not found
Error relocating /usr/local/bin/node: _ZNSo3putEc: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE7seekoffElSt12_Ios_SeekdirSt13_Ios_Openmode: symbol not found
Error relocating /usr/local/bin/node: __cxa_guard_release: symbol not found
Error relocating /usr/local/bin/node: _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE7seekposESt4fposI11__mbstate_tESt13_Ios_Openmode: symbol not found
Error relocating /usr/local/bin/node: _ZNSi10_M_extractIdEERSiRT_: symbol not found
Error relocating /usr/local/bin/node: _ZSt20__throw_length_errorPKc: symbol not found
Error relocating /usr/local/bin/node: _ZSt20__throw_out_of_rangePKc: symbol not found
Error relocating /usr/local/bin/node: __popcountdi2: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8ios_base4InitD1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt9basic_iosIcSt11char_traitsIcEE5rdbufEPSt15basic_streambufIcS1_E: symbol not found
Error relocating /usr/local/bin/node: __register_atfork: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE9showmanycEv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt18condition_variableD1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8ios_baseD2Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNKSt8__detail20_Prime_rehash_policy11_M_next_bktEm: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertIdEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZNSt6localeC1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt10_Sp_lockerD1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl: symbol not found
Error relocating /usr/local/bin/node: _ZNSt13basic_filebufIcSt11char_traitsIcEEC1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt18condition_variableC1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_: symbol not found
Error relocating /usr/local/bin/node: _ZNSo5seekpElSt12_Ios_Seekdir: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE9underflowEv: symbol not found
Error relocating /usr/local/bin/node: _ZSt18_Rb_tree_incrementPKSt18_Rb_tree_node_base: symbol not found
Error relocating /usr/local/bin/node: setcontext: symbol not found
Error relocating /usr/local/bin/node: __cxa_pure_virtual: symbol not found
Error relocating /usr/local/bin/node: _ZNSt18condition_variable10notify_allEv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8__detail15_List_node_base9_M_unhookEv: symbol not found
Error relocating /usr/local/bin/node: __dynamic_cast: symbol not found
Error relocating /usr/local/bin/node: _ZNSt13basic_filebufIcSt11char_traitsIcEE4openEPKcSt13_Ios_Openmode: symbol not found
Error relocating /usr/local/bin/node: _ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base: symbol not found
Error relocating /usr/local/bin/node: _ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8ios_baseC2Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt6localeD1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNKSt8__detail20_Prime_rehash_policy14_M_need_rehashEmmm: symbol not found
Error relocating /usr/local/bin/node: _ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_: symbol not found
Error relocating /usr/local/bin/node: _ZNSt8ios_base4InitC1Ev: symbol not found
Error relocating /usr/local/bin/node: _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE: symbol not found
Error relocating /usr/local/bin/node: _ZNKSt12__basic_fileIcE7is_openEv: symbol not found
Error relocating /usr/local/bin/node: _ZnwmRKSt9nothrow_t: symbol not found
Error relocating /usr/local/bin/node: makecontext: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE5uflowEv: symbol not found
Error relocating /usr/local/bin/node: _ZNKSt9basic_iosIcSt11char_traitsIcEE5widenEc: symbol not found
Error relocating /usr/local/bin/node: _ZSt18_Rb_tree_decrementPKSt18_Rb_tree_node_base: symbol not found
Error relocating /usr/local/bin/node: _ZnamRKSt9nothrow_t: symbol not found
Error relocating /usr/local/bin/node: _ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_: symbol not found
Error relocating /usr/local/bin/node: _ZdlPvRKSt9nothrow_t: symbol not found
Error relocating /usr/local/bin/node: _ZSt25__throw_bad_function_callv: symbol not found
Error relocating /usr/local/bin/node: _ZNSt9basic_iosIcSt11char_traitsIcEE5clearESt12_Ios_Iostate: symbol not found
Error relocating /usr/local/bin/node: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1ESt13_Ios_Openmode: symbol not found
Error relocating /usr/local/bin/node: backtrace: symbol not found
Error relocating /usr/local/bin/node: getcontext: symbol not found
Error relocating /usr/local/bin/node: _ZNSt15basic_streambufIcSt11char_traitsIcEE6setbufEPcl: symbol not found
Error relocating /usr/local/bin/node: _ZNSo9_M_insertIxEERSoT_: symbol not found
Error relocating /usr/local/bin/node: _ZSt11__once_call: symbol not found
Error relocating /usr/local/bin/node: _ZSt15__once_callable: symbol not found
Error relocating /usr/local/bin/node: _ZSt7nothrow: symbol not found
Error relocating /usr/local/bin/node: _ZTVN10__cxxabiv121__vmi_class_type_infoE: symbol not found
Error relocating /usr/local/bin/node: _ZTVNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTTSt14basic_ofstreamIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating /usr/local/bin/node: _ZTVNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVSt14basic_ifstreamIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVNSt7__cxx1119basic_istringstreamIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTTNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/bin/node: _ZTVSt15basic_streambufIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: _ZTTNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVSt14basic_ofstreamIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: _ZTTSt14basic_ifstreamIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: __libc_stack_end: symbol not found
Error relocating /usr/local/bin/node: _ZTVSt9basic_iosIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: _ZTTNSt7__cxx1119basic_istringstreamIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/bin/node: _ZTVSt13basic_filebufIcSt11char_traitsIcEE: symbol not found
Error relocating /usr/local/bin/node: _ZSt4cout: symbol not found
Error relocating /usr/local/bin/node: _ZSt4cerr: symbol not found

Answer 1

Thanks for including the ldd output. It clearly tells you what's wrong. The /usr/local/bin/node binary depends on shared libraries which are missing from the target system.

https://packages.debian.org/ lets you search for file names (scroll down to "search package contents" near the bottom); generally you want to skip everything after .so in the search when looking for shared libraries.

The search results (linked below) are kind of intimidating. You want to ignore all the cross-compilation stuff; my money would be on the below list, but ideally the people you got the binary from should be able to give authoritative information.

• libgcc-12-dev ... or maybe libgcc-11-dev or etc
• libstdc++-12-dev ... or maybe libstdc++11-dev or etc

There could be additional downstream problems if you resolve these two, so you may get a new error and then need to iterate over that.