Load different sweep files in Hydra multirun

Question

├── configs
│   ├── default.yaml
│   └── agent
│       └── agent_A.yaml
│   └── experiment
│       └── long.yaml
│       └── short.yaml
│   └── environment
│       └── easy.yaml
│       └── hard.yaml
│   └── hydra
│       └── launcher
│           └── cluster_X.yaml
│           └── cluster_Y.yaml

I need to perform different sweeps. Right now, I do it by adding the following config to default.yaml

hydra:
  sweeper:
    params:
      environment: easy
      experiment: short
      experiment.rng_seed: range(0, 10)
      agent.lr: 1e-3, 1e-2, 1e-1
      agent.size: 64, 128, 256

My problem is that I need to edit the file every time I need to run a sweep because the combinations I need to run are:

• environment easy / experiment short
• environment hard / experiment long And I may also want to sweep over a different range of agent.lr and agent.size depending on the sweep.

I run my code with

python main.py -m hydra/launcher=cluster_X 

and I would like to add something like

python main.py -m hydra/launcher=cluster_X  hydra/sweeper=sweep_1

where sweep_1.yaml is in hydra/sweeper and is a config file with the details of the sweep. Then I'd have many sweep_1, sweep_2, ... files for whatever I need to run, not having to edit default.yaml every time.

I tried making sweep_1.yaml like this

# @package hydra.sweeper
params:
  environment: easy
  experiment: short
  experiment.rng_seed: range(0, 100)
  agent.lr: 0.1, 1

but I don't know what _target_ use. I am not even sure this is the right way to do what I need to do.

Answer 1

By changing my sweeper YAML files to something like this I can do exactly what I wanted to.

# @package _global_
# @package hydra.sweeper
_target_: hydra._internal.core_plugins.basic_sweeper.BasicSweeper
max_batch_size: null
params:
  environment: easy
  experiment: short
  experiment.rng_seed: range(0, 100)
  agent.lr: 0.1, 1

This uses the default sweeper. max_batch_size is mandatory. .