I am trying to launch a gradio backend but I seem to be missing libraries, although I have them downloaded.
(.venv) reply@reply-GP66-Leopard-11UH:~/Documents/chatbot-rag$ python3 gradio-chatbot.py
/home/reply/Documents/chatbot-rag/.venv/lib/python3.10/site-packages/langchain/__init__.py:34: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain.prompts.PromptTemplate instead.
warnings.warn(
Initializing backend for chatbot
/home/reply/Documents/chatbot-rag/.venv/lib/python3.10/site-packages/transformers/pipelines/__init__.py:698: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
Traceback (most recent call last):
File "/home/reply/Documents/chatbot-rag/gradio-chatbot.py", line 10, in <module>
backend.load_embeddings_and_llm_models()
File "/home/reply/Documents/chatbot-rag/backend.py", line 50, in load_embeddings_and_llm_models
self.llm = self.load_llm(self.params)
File "/home/reply/Documents/chatbot-rag/backend.py", line 66, in load_llm
pipe = pipeline("text-generation", model=self.llm_name_or_path, model_kwargs=model_kwargs)
File "/home/reply/Documents/chatbot-rag/.venv/lib/python3.10/site-packages/transformers/pipelines/__init__.py", line 870, in pipeline
framework, model = infer_framework_load_model(
File "/home/reply/Documents/chatbot-rag/.venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 269, in infer_framework_load_model
model = model_class.from_pretrained(model, **kwargs)
File "/home/reply/Documents/chatbot-rag/.venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
return model_class.from_pretrained(
File "/home/reply/Documents/chatbot-rag/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2714, in from_pretrained
raise ImportError(
ImportError: Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or pip install bitsandbytes`
But I already have them. And they are up to date:
(.venv) reply@reply-GP66-Leopard-11UH:~/Documents/chatbot-rag$ pip freeze | grep -E '^(accelerate|bitsandbytes)'
accelerate==0.24.1
bitsandbytes==0.41.1
(.venv) reply@reply-GP66-Leopard-11UH:~/Documents/chatbot-rag$ pip install --upgrade accelerate
pip install --upgrade -i https://test.pypi.org/simple/ bitsandbytes
Requirement already satisfied: accelerate in ./.venv/lib/python3.10/site-packages (0.24.1)
Requirement already satisfied: numpy>=1.17 in ./.venv/lib/python3.10/site-packages (from accelerate) (1.26.1)
Requirement already satisfied: packaging>=20.0 in ./.venv/lib/python3.10/site-packages (from accelerate) (23.2)
Requirement already satisfied: psutil in ./.venv/lib/python3.10/site-packages (from accelerate) (5.9.6)
Requirement already satisfied: pyyaml in ./.venv/lib/python3.10/site-packages (from accelerate) (6.0.1)
Requirement already satisfied: torch>=1.10.0 in ./.venv/lib/python3.10/site-packages (from accelerate) (2.1.0)
Requirement already satisfied: huggingface-hub in ./.venv/lib/python3.10/site-packages (from accelerate) (0.17.3)
Requirement already satisfied: filelock in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.13.1)
Requirement already satisfied: typing-extensions in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (4.8.0)
Requirement already satisfied: sympy in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (1.12)
Requirement already satisfied: networkx in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.2.1)
Requirement already satisfied: jinja2 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.1.2)
Requirement already satisfied: fsspec in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (2023.10.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (2.18.1)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105)
Requirement already satisfied: triton==2.1.0 in ./.venv/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (2.1.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in ./.venv/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate) (12.3.52)
Requirement already satisfied: requests in ./.venv/lib/python3.10/site-packages (from huggingface-hub->accelerate) (2.31.0)
Requirement already satisfied: tqdm>=4.42.1 in ./.venv/lib/python3.10/site-packages (from huggingface-hub->accelerate) (4.66.1)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.3)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate) (2023.7.22)
Requirement already satisfied: mpmath>=0.19 in ./.venv/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)
Looking in indexes: https://test.pypi.org/simple/
Requirement already satisfied: bitsandbytes in ./.venv/lib/python3.10/site-packages (0.41.1)
The interpreter is that of virtualenv:
(.venv) reply@reply-GP66-Leopard-11UH:~/Documents/chatbot-rag$ which python
/home/reply/Documents/chatbot-rag/.venv/bin/python
(.venv) reply@reply-GP66-Leopard-11UH:~/Documents/chatbot-rag$ which python3
/home/reply/Documents/chatbot-rag/.venv/bin/python3
It seems like there is an issue with transformers version.
Installing the version 4.37 fixed the issue for me.
pip install transformers==4.37