using hydra with pyinstaller

Question

I have been trying to use Hydra with PyInstaller and failed. I have created a very configuration example similar to example in here.

I noticed that hydra packages are not being found by PyInstaller so I created a simple hook file hook-hydra.py with the following code:

from PyInstaller.utils.hooks import collect_data_files, collect_submodules

datas = collect_data_files('hydra')
hiddenimports = collect_submodules('hydra')

That seemed to solve the module imports failures, but then when I tried to run the executable in the command line I got the following error:

Traceback (most recent call last):
  File "lib\site-packages\hydra\_internal\utils.py", line 198, in run_and_report
  File "lib\site-packages\hydra\_internal\utils.py", line 321, in <lambda>
  File "lib\site-packages\hydra\_internal\hydra.py", line 74, in create_main_hydra2
  File "lib\site-packages\hydra\_internal\config_loader_impl.py", line 80, in __init__
  File "lib\site-packages\hydra\_internal\config_repository.py", line 22, in __init__
  File "lib\site-packages\hydra\_internal\sources_registry.py", line 30, in resolve
ValueError: No config source registered for schema pkg, supported types : []

I can't seem to figure it out, any ideas?
I'm using PyInstaller 3.6 and Hydra 1.0.4

Answer 1

After taking a look at PyInstaller, it looks like it's attempting to discover needed packages and it somehow fails to do a good job for Hydra. Hydra has some built in plugins that are discovered at runtime, including the config sources. The error suggests that the config sources were not packages by PyInstaller.

If PyInstaller is trying to be clever and only include things it sees a direct dependency on it is likely to fail for Hydra. Try to add all hydra modules explicitly to your PyInstaller config file.

As an alternative packaging method for your app, take a look at the application packaging example here. It shows you the supported method for installing Hydra apps (with a working example).

Answer 2

Hydra does not play nice with PyInstaller because it dynamically reads the plugin packages using pkgutil.walk_packages.

Check out the issue and answers here: Pyinstaller - include programmatically imported modules

As you are not solving this from a library developer's point of view, copying the package using the data field in the .spec file could solve your issue, although it is a bit hacky. Either this or switching to OmegaConf is a good solution in my opinion.

Answer 3

This works for me, no problems running from command line, using PyInstaller 4.5.1, Hydra 1.2:

from PyInstaller.utils.hooks import collect_data_files, collect_submodules

datas = collect_data_files('hydra', subdir='conf')
hiddenimports = collect_submodules('hydra')

Answer 4

I faced similar issues and ended up having to create a pyinstaller hook named hook-hydra.py and put it in the pyinstaller hooks folder. It could probably be reduced to lower level hydra namespaces. Leveraging other hooks, I did the following:

Only issue I face is inline using compose overrides=[f"environment.root_path={set_root_path}"], complains about EOF .... etc. But have workaround for that. Logging and Fetching configuration works like a charm.

from PyInstaller.utils.hooks import collect_submodules

#  Use this to get hydra internals
#  "hydra._internal.core_plugins"
# "hydra.plugins"
# 'hydra._internal.core_plugins','hydra.grammar.gen.OverrideParser'

hiddenimports = collect_submodules('hydra')