PyInstaller Compiled Application cannot not find Specific Resources from sys.path

Question

When trying to create a plugin feature for a larger application I came across an issue with the main.exe generated using PyInstaller on Windows (I tried with Python 3.8 (x86) and Python 3.11 and PyInstaller 5.11.0).

The goal is to create an application that can be distributed as a package (folder) created by PyInstaller and provides a mechanism to import plugin scripts at runtime. The target machines will have a system wide Python installation, with the same version as the application was developed and compiled with. Since one cannot know in advance, which modules those plugins will need to import so that the --hidden-import cannot be used at compilation time, the application will expand the sys.path variable to include the folders of the packages of the system wide Python installation folder.

The boiled-down version of the main.py file would like like:

    import sys
    import os
    import importlib
    
    # import xmlrpc.client          # activate later on, compile, execute and compare the error messages...
    # print(dir(xmlrpc.client)[0:3])
    
    # PATH_TO_PYTHON_INSTALLATION = r'C:\Program Files (x86)\Python38'
    PATH_TO_PYTHON_INSTALLATION = r'C:\Program Files\Python311'
    PYTHONPATH = os.environ.get('PYTHONPATH', PATH_TO_PYTHON_INSTALLATION)
    LIB_FOLDERS = ['DLLs', 'Lib', '', 'Lib\\site-packages']
    for folder in LIB_FOLDERS:
        sys.path.append(os.path.join(PYTHONPATH, folder))
        pass
    
    plugin = importlib.import_module('plugin')
    print(dir(plugin.xmlrpc.server)[0:3])

For testing a short plugin.py would do:

    import xmlrpc.server
    print(dir(xmlrpc.server)[0:3])

Starting the application with the interpreter, everything works as expected.

But when the application is compiled into a dist/main/main.exe executable, the result will be:

Traceback (most recent call last):
  File "main.py", line 13, in <module>
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\pyinstaller_issue\dist\main\plugin.py", line 1, in <module>
    import xmlrpc.server
  File "C:\Program Files\Python311\Lib\xmlrpc\server.py", line 108, in <module>
    from http.server import BaseHTTPRequestHandler
ModuleNotFoundError: No module named 'http.server'
[29388] Failed to execute script 'main' due to unhandled exception!

The error indicates that the xmlrpc.server module was loaded from the system wide installation, but the http.server could not be imported.

It is becoming even more confusing, when one activates the import xmlrpc.client statement in the main.py, compiles again and reruns the exe-file:

['APPLICATION_ERROR', 'Binary', 'Boolean']
Traceback (most recent call last):
  File "main.py", line 16, in <module>
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\pyinstaller_issue\dist\main\plugin.py", line 1, in <module>
    import xmlrpc.server
ModuleNotFoundError: No module named 'xmlrpc.server'
[24332] Failed to execute script 'main' due to unhandled exception!

I understand that in the second example the xmlrpc.client was found as an import by the pyinstaller tool and is therefore included as a frozen import into the distribution package.

What I do not understand, why now the xmlrpc.server package cannot even be found anymore (which it could before)?

Edit 2023-05-26

After having posted the question here I learned about the discussion site on PyInstaller at github and posted the question there: https://github.com/orgs/pyinstaller/discussions/7645

Thanks to the hints by Rok Mandeljc I was able to create a little package that can compare the modules that would be frozen against the ones available in the standard Python installation and iteratively extends the hidden-import list times until every submodule of all packages are included if there was a partial import already.

So I decided to create a helping package (https://pypi.org/project/wrapenv/) to be able to tweak the Analysis() step of the build process as shown in the example file: https://github.com/AMueckl/WrapEnv/blob/main/examples/run_pyinstaller.py

Using the extended build process the application now allows to import from packages that would not have been available with the standard PyInstaller process.

For more information see Demonstration of Use of the WrapEnv Package.