Error opening shared object: how do I embed shared object or specify a different pathname programmatically?

Question

So, I am deploying lisp image to server for a webapp. It has been a nice experience so far, except for this particular case:

I am generating the webapp using this:

sbcl --dynamic-space-size 1024 \
     --noinform \
     --load $HOME/quicklisp/setup.lisp \
     --eval '(ql:quickload :webapp)' \
     --eval '(swank-loader:init :load-contribs t)' \
     --eval "(sb-ext:save-lisp-and-die \"webapp\" :toplevel #'webapp::executable-entry-point :executable t :compression t)"

However, when I start the webapp on the server, I am thrown into the debugger:

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10006285B3}>:
  Error opening shared object "/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so":
  /home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so: cannot open shared object file: No such file or directory.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE       ] Skip this shared object and continue.
  1: [RETRY          ] Retry loading this shared object.
  2: [CHANGE-PATHNAME] Specify a different pathname to load the shared object from.
  3: [ABORT          ] Exit from the current thread.

(SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so" :NAMESTRING "/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so" :HANDLE NIL :DONT-SAVE NIL))
0] back

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10006285B3}>
0: (SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so" :NAMESTRING "/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so" :HANDLE NIL :DONT-SAVE NIL))
1: (SB-ALIEN::TRY-REOPEN-SHARED-OBJECT #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so" :NAMESTRING "/home/username/quicklisp/dists/quicklisp/software/clsql-20160208-git/db-mysql/clsql_mysql64.so" :HANDLE NIL :DONT-SAVE NIL))
2: (SB-SYS:REOPEN-SHARED-OBJECTS)
3: (SB-IMPL::FOREIGN-REINIT)
4: (SB-IMPL::REINIT)
5: ((FLET SB-UNIX::BODY :IN SAVE-LISP-AND-DIE))
6: ((FLET "WITHOUT-INTERRUPTS-BODY-36" :IN SAVE-LISP-AND-DIE))
7: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))

0] 3

I'm not sure if this is limited to clsql - is there a way to embed the shared object or specify the pathname programmatically, since this looks like the error occurs even before the toplevel function is even called.

There does exist one two year old discussion about embedding shared object files into lisp images - the recommendation was to go with ECL, at least until ASDF instructions are clear.

For SBCL, I also discovered sb-alien:load-shared-object with the :dont-save nil option; but not sure how to put it to use here.

Answer 1

In the general case, deploy should do the trick. (Thanks to u/flaming_bird for pointing it out here.)

However, in this particular case, clsql looks for the .so files in clsql-sys:*foreign-library-search-paths*. This "lookup" happens during the compilation / make. Once done, the path that was used gets hardcoded into the image, and an error will be thrown if the library is not at that exact same path.

An alternative is to delete (or rename) the .so file from the clsql installation directory ($HOME/quicklisp/software/clsql-20160208-git/db-mysql/ in this case) before the image build and instead put it in the place (say, the current directory #P"./") where it is needed in the deployment server. And then, modifying the build command to push the shared-library location (eg. #P"./") does the trick:

sbcl --dynamic-space-size 1024 \
     --noinform \
     --load $HOME/quicklisp/setup.lisp \
     --eval '(ql:quickload :clsql)' \
     --eval '(push #P"./" clsql-sys:*foreign-library-search-paths*)' \
     --eval '(ql:quickload :webapp)' \
     --eval '(swank-loader:init :load-contribs t)' \
     --eval '(py4cl2:pystop)' \
     --eval "(sb-ext:save-lisp-and-die \"webapp\" :toplevel #'webapp::executable-entry-point :executable t :compression t)"