How to use hydra compose API to read yaml from a nested directory?

Question

I am trying to print the src.yaml from a nested directory below

Main_dir:
 conf:
  project:
    test_project1:
      env:
        source:
            src.yaml

                 

src.yaml

table: TEST_TABLE
DB:Test_DB

I tried the following hydra API

with initialize(version_base=None, config_path="../../conf"):
 cfg = compose(config_name="config", overrides = ["project=test_project1","+source=src"])
 print(cfg)

but I am getting the following error:

hydra.errors.ConfigCompositionException: Could not override 'project'. No match in the default list

How can I get rid to of this error?

Answer 1

project=test_project1 means that there is a CONFIG named test_project1. Not a directory.

The Compose API gives you the same config composition capabilities @hydra.main() gives you. What you are trying to do is not supported in that form.

Something like this is more likely to work:

with initialize(version_base=None, config_path="../../conf"):
 cfg = compose(config_name="config", overrides = [
  "project/test_project1/env/source=src"
 ])
 print(cfg)

You are providing the conf config directory as your config path. This means all the sub directories are considered config groups.

I suggest you go back and read the Hydra basic tutorial. If you want to get deeper, I recommend The Defaults List.