Passing multiple config groups

Question

In my config.yaml, how to pass two datasets for e.g. cifar and cinic at once? Can I pass a multiple config groups to the defaults list?

This is for the case when I want to train my model on a mix of datasets, but I do not want to create a config group for every possible combination.

├── config.yaml
└── dataset
    ├──  cifar.yaml
    └── imagenet.yaml
    └── cinic.yaml

What I tried is as follows:

dataset: 
  - cifar
  - cinic

which resulted in following error:

Could not load train_dataset/['cifar', 'cinic']. Available options 'cifar ... '

Answer 1

Currently config groups are mutually exclusive. Support for this is planned for Hydra 1.1. See issue 499.

One possible workaround is to put everything in the config and to use interpolation:

all_datasets:
  imagenet:
    name: imagenet
  cifar10:
    name: cifar10

datasets:
  - ${all_datasets.imagenet}
  - ${all_datasets.cifar10}

This way you can override dataset to a different list of datasets from the command line (with the interpolation).

If you want to simplify the usage at the expense of some additional code, you can do something like:

all_datasets:
  ...

datasets_list:
  - imagenet
  - cifar10

datasets: []

@hydra.main(config_path="conf", config_name="config")
def my_app(cfg: DictConfig) -> None:
  for ds in cfg.datasets_list:
    cfg.datasets.append(cfg.all_datasets[ds])

if __name__ == "__main__":
    my_app()

I didn't test this but I hope you get the idea.