RuntimeError: Failed to import transformers.models.clip.processing_clip because of the following error

Question

RuntimeError: Failed to import transformers.models.clip.processing_clip because of the following error (look up to see its traceback):
[Errno 2] No such file or directory: 'C:\\Users\\xxx\\AppData\\Local\\Temp\\_MEI135322\\transformers\\__init__.py'

I am trying to create an .exe file using pyinstaller of a script that has the sentence_transformers library.

This is the one line command (adjust for readability) that I use to run pyinstaller:

pyinstaller --onefile \
            --hidden-import=torch \
            --collect-data torch \
            --copy-metadata torch \
            --collect-data tqdm \
            --copy-metadata tqdm \
            --collect-data regex \
            --copy-metadata regex \
            --collect-data sacremoses \
            --copy-metadata sacremoses \
            --collect-data requests \
            --copy-metadata requests \
            --collect-data packaging \
            --copy-metadata packaging \
            --collect-data filelock \
            --copy-metadata filelock \
            --collect-data numpy \
            --copy-metadata numpy \
            --collect-data tokenizers \
            --copy-metadata tokenizers \
            --collect-data importlib_metadata \
            --copy-metadata importlib_metadata \
            --hidden-import="sklearn" \
            --copy-metadata huggingface-hub \
            --copy-metadata safetensors \
            --copy-metadata PyYAML \
            --hidden-import="sklearn.neighbors._typedefs" \
            --hidden-import="matplotlib" \
            --additional-hooks-dir=. \
            my_script.py

This is my first time using pyinstaller and I can't find anymore resources for this error and I hope I could get some leads from here. Do let me know if I'm missing any important information to include in this question. Appreciate any advice given. Thank you.

Answer 1

Edited my question above to keep it short. But these are my findings and solution.

The _MEI135322 folder is a temp file that is created by pyinstaller with all the needed files to run the exe file. Looking through the folder during running the executable, I found there is no "transformers" file in the folder.

Here's what I did:

1. Double checked if I have sentence-transformers and transformers installed by using "pip list" in cmd.

2. Get the path of where these modules were installed by using "pip show transformers" or "pip show sentence-transformers" in cmd.

3. With the full path of to the module found, add on --add-data "<path_to_transformers>;transformers" or --add-data "<path_to_sentence_transformers>;sentence_transformers" to the pyinstaller command.

I did use --hidden-import before this, but I'm not sure why it didn't include it during packaging. The above steps fixed it for me by directing pyinstaller straight to the location of the modules. Hope this helps.

Answer 2

I am using this command to solve this

pyinstaller --noconsole --hidden-import=torch --collect-data torch --copy-metadata torch --copy-metadata tqdm --copy-metadata regex --copy-metadata sacremoses --copy-metadata requests --copy-metadata packaging --copy-metadata filelock --copy-metadata numpy --copy-metadata tokenizers --copy-metadata importlib_metadata --hidden-import="sklearn.utils._cython_blas" --hidden-import="sklearn.neighbors.typedefs" --hidden-import="sklearn.neighbors.quad_tree" --hidden-import="sklearn.tree" --hidden-import="sklearn.tree._utils" --hidden-import=huggingface_hub --hidden-import=safetensors --copy-metadata=safetensors --copy-metadata huggingface_hub --hidden-import=yaml --copy-metadata=pyyaml --hidden-import=transformers --collect-submodules transformers -w --exclude-module onnxscript --exclude-module sklearn.neighbors._typedefs main.py
)