I'm facing trouble dockerizing an LLM based application that I'm working on. I have a python/chainlit app that runs fine when I start it through the terminal, however, it seems to fail when I try to run it through a docker container.
Meta information:
Python Version: 3.11.4
Docker version: 24.0.6, build ed223bc
Operating System: MacOS 13.6.2 (22G320)
Chip: Apple M1 Pro
Locally, I run things as follows:
Create python virtual environment
python3 -m venv env
source env/bin/activate
Run a pip install of all necessary libraries
pip install -r requirements.txt
Start application
chainlit run app.py -w
The requirements.txt file has the following:
accelerate==0.24.1
aiofiles==23.2.1
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==3.7.1
appnope==0.1.3
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
asyncer==0.0.2
attrs==23.1.0
Babel==2.13.1
backoff==2.2.1
beautifulsoup4==4.12.2
bidict==0.22.1
bitsandbytes==0.41.2.post2
bleach==6.1.0
certifi==2023.11.17
cffi==1.16.0
chainlit==0.7.700
charset-normalizer==2.0.12
click==8.1.7
comm==0.2.0
ctransformers==0.2.27
dataclasses-json==0.5.14
debugpy==1.8.0
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.14
executing==2.0.1
faiss-cpu==1.7.4
fastapi==0.100.1
fastapi-socketio==0.0.10
fastjsonschema==2.19.0
filelock==3.13.1
filetype==1.2.0
fqdn==1.5.1
frozenlist==1.4.0
fsspec==2023.10.0
googleapis-common-protos==1.61.0
grpcio==1.59.3
h11==0.14.0
httpcore==0.17.3
httpx==0.24.1
huggingface-hub==0.19.4
idna==3.6
importlib-metadata==6.8.0
ipykernel==6.27.1
ipython==8.18.1
ipywidgets==8.1.1
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.2
joblib==1.3.2
json5==0.9.14
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.9.0
jupyter-lsp==2.2.1
jupyter_client==8.6.0
jupyter_core==5.5.0
jupyter_server==2.11.1
jupyter_server_terminals==0.4.4
jupyterlab==4.0.9
jupyterlab-widgets==3.0.9
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.2
langchain==0.0.343
langchain-core==0.0.7
langsmith==0.0.67
Lazify==0.4.0
MarkupSafe==2.1.3
marshmallow==3.20.1
matplotlib-inline==0.1.6
mistune==3.0.2
mpmath==1.3.0
multidict==6.0.4
mypy-extensions==1.0.0
nbclient==0.9.0
nbconvert==7.11.0
nbformat==5.9.2
nest-asyncio==1.5.8
networkx==3.2.1
nltk==3.8.1
notebook==7.0.6
notebook_shim==0.2.3
numpy==1.26.2
opentelemetry-api==1.21.0
opentelemetry-exporter-otlp==1.21.0
opentelemetry-exporter-otlp-proto-common==1.21.0
opentelemetry-exporter-otlp-proto-grpc==1.21.0
opentelemetry-exporter-otlp-proto-http==1.21.0
opentelemetry-instrumentation==0.42b0
opentelemetry-proto==1.21.0
opentelemetry-sdk==1.21.0
opentelemetry-semantic-conventions==0.42b0
overrides==7.4.0
packaging==23.2
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.9.0
Pillow==10.1.0
platformdirs==4.0.0
prometheus-client==0.19.0
prompt-toolkit==3.0.41
protobuf==4.25.1
psutil==5.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
pycparser==2.21
pydantic==2.5.2
pydantic_core==2.14.5
Pygments==2.17.2
PyJWT==2.8.0
pypdf==3.17.1
python-dateutil==2.8.2
python-dotenv==1.0.0
python-engineio==4.8.0
python-graphql-client==0.4.3
python-json-logger==2.0.7
python-multipart==0.0.6
python-socketio==5.10.0
PyYAML==6.0.1
pyzmq==25.1.1
qtconsole==5.5.1
QtPy==2.4.1
referencing==0.31.1
regex==2023.10.3
requests==2.31
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.13.2
safetensors==0.4.1
scikit-learn==1.3.2
scipy==1.11.4
Send2Trash==1.8.2
sentence-transformers==2.2.2
sentencepiece==0.1.99
simple-websocket==1.0.0
six==1.16.0
sniffio==1.3.0
soupsieve==2.5
SQLAlchemy==2.0.23
stack-data==0.6.3
starlette==0.27.0
sympy==1.12
syncer==2.0.3
tenacity==8.2.3
terminado==0.18.0
threadpoolctl==3.2.0
tinycss2==1.2.1
tokenizers==0.15.0
tomli==2.0.1
torch==2.1.1
torchvision==0.16.1
tornado==6.4
tqdm==4.66.1
traitlets==5.14.0
transformers==4.35.2
types-python-dateutil==2.8.19.14
typing-inspect==0.9.0
typing_extensions==4.8.0
uptrace==1.21.0
uri-template==1.3.0
urllib3==1.26.18
uvicorn==0.23.2
watchfiles==0.20.0
wcwidth==0.2.12
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.4
websockets==12.0
widgetsnbextension==4.0.9
wrapt==1.16.0
wsproto==1.2.0
yarl==1.9.3
zipp==3.17.0
With Docker, I run the following commands:
Build Docker image
docker build -t local:app .
The Docker File is as follows:
FROM python:3.11.4
WORKDIR /app
RUN pip install --upgrade pip
COPY /src /app
RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
EXPOSE 8000
CMD ["chainlit", "run", "app.py", "-w"]
Start Docker container
docker run -p 8000:8000 local:app
The image builds successfully, and the application even starts, but I get the following error log:
/usr/local/lib/python3.11/site-packages/langchain/__init__.py:34: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain.prompts.PromptTemplate instead.
warnings.warn(
2023-12-02 02:29:46 - Your app is available at http://localhost:8000
2023-12-02 02:29:53 - Load pretrained SentenceTransformer: ./all-MiniLM-L6-v2
2023-12-02 02:29:53 - Loading faiss.
2023-12-02 02:29:53 - Successfully loaded faiss.
2023-12-02 02:29:53 - /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/chainlit/utils.py", line 39, in wrapper
return await user_function(**params_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app.py", line 11, in start
chain = qa_bot()
^^^^^^^^
File "/app/model.py", line 64, in qa_bot
llm = load_llm()
^^^^^^^^^^
File "/app/model.py", line 48, in load_llm
llm = CTransformers(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 97, in __init__
super().__init__(**kwargs)
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 339, in __init__
values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 1102, in validate_model
values = validator(cls_, values)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain/llms/ctransformers.py", line 73, in validate_environment
values["client"] = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/hub.py", line 175, in from_pretrained
llm = LLM(
^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 246, in __init__
self._lib = load_library(lib, gpu=config.gpu_layers > 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 126, in load_library
lib = CDLL(path)
^^^^^^^^^^
File "/usr/local/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
2023-12-02 02:29:53 - Load pretrained SentenceTransformer: ./all-MiniLM-L6-v2
2023-12-02 02:29:53 - /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/chainlit/utils.py", line 39, in wrapper
return await user_function(**params_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app.py", line 11, in start
chain = qa_bot()
^^^^^^^^
File "/app/model.py", line 64, in qa_bot
llm = load_llm()
^^^^^^^^^^
File "/app/model.py", line 48, in load_llm
llm = CTransformers(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 97, in __init__
super().__init__(**kwargs)
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 339, in __init__
values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 1102, in validate_model
values = validator(cls_, values)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain/llms/ctransformers.py", line 73, in validate_environment
values["client"] = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/hub.py", line 175, in from_pretrained
llm = LLM(
^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 246, in __init__
self._lib = load_library(lib, gpu=config.gpu_layers > 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 126, in load_library
lib = CDLL(path)
^^^^^^^^^^
File "/usr/local/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
2023-12-02 02:29:53 - Load pretrained SentenceTransformer: ./all-MiniLM-L6-v2
2023-12-02 02:29:54 - /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/chainlit/utils.py", line 39, in wrapper
return await user_function(**params_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app.py", line 11, in start
chain = qa_bot()
^^^^^^^^
File "/app/model.py", line 64, in qa_bot
llm = load_llm()
^^^^^^^^^^
File "/app/model.py", line 48, in load_llm
llm = CTransformers(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 97, in __init__
super().__init__(**kwargs)
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 339, in __init__
values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 1102, in validate_model
values = validator(cls_, values)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain/llms/ctransformers.py", line 73, in validate_environment
values["client"] = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/hub.py", line 175, in from_pretrained
llm = LLM(
^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 246, in __init__
self._lib = load_library(lib, gpu=config.gpu_layers > 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 126, in load_library
lib = CDLL(path)
^^^^^^^^^^
File "/usr/local/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
2023-12-02 02:29:54 - Load pretrained SentenceTransformer: ./all-MiniLM-L6-v2
2023-12-02 02:29:54 - /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/chainlit/utils.py", line 39, in wrapper
return await user_function(**params_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app.py", line 11, in start
chain = qa_bot()
^^^^^^^^
File "/app/model.py", line 64, in qa_bot
llm = load_llm()
^^^^^^^^^^
File "/app/model.py", line 48, in load_llm
llm = CTransformers(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 97, in __init__
super().__init__(**kwargs)
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 339, in __init__
values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 1102, in validate_model
values = validator(cls_, values)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain/llms/ctransformers.py", line 73, in validate_environment
values["client"] = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/hub.py", line 175, in from_pretrained
llm = LLM(
^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 246, in __init__
self._lib = load_library(lib, gpu=config.gpu_layers > 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ctransformers/llm.py", line 126, in load_library
lib = CDLL(path)
^^^^^^^^^^
File "/usr/local/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.11/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
I suspect it has something to do with the ctransformers library. Additionally, I ssh'ed into the container and found that the file libctransformers.so is in fact present and not missing.