Use parameters from additional configs in dvc 2.0

Question

Using dvc version 2.0.18 and python 3.9.2 I want to use parameters defined in a config file different from params.yaml when configuring the parameters of the stages in dvc.yaml. However, it does not work as I expected.

MWE: Git repo + dvc init:

.
├── dvc.yaml
├── preproc.yaml
└── test.py

dvc.yaml:

vars:
  - preproc.yaml
stages:
  test:
    cmd: python test.py
    deps:
      - test.py
    params:
      - important_parameter

preproc.yaml:

important_parameter: 123

Running dvc repro lead to the following error:

ERROR: failed to reproduce 'dvc.yaml': dependency 'params.yaml' does not exist

Creating a dummy params.yaml without content gives:

WARNING: 'params.yaml' is empty.
ERROR: failed to reproduce 'dvc.yaml': Parameters 'important_parameter' are missing from 'params.yaml'.

What am I missing? Is this possible at all with the templating feature?

Answer 1

I think you don't need the templating feature in this case. As shown in this example:

stages:
  train:
    cmd: python train.py
    deps:
      - users.csv
    params:
      - params.py:
          - BOOL
          - INT
          - TrainConfig.EPOCHS
          - TrainConfig.layers
    outs:
      - model.pkl

The way to redefine the default params.yaml is to specify the file name explicitly in the params: section:

params:
  - preproc.yaml:
    - important_parameter

Also, when you create a stage either with dvc run (not recommended) or dvc stage add, you can provide the params file name explicitly as a prefix:

dvc run -n train -d train.py -d logs/ -o users.csv -f \
          -p parse_params.yaml:threshold,classes_num \
          python train.py

Here ^^ parse_params.yaml is a custom params file.

Please, let me know if it solves the problem and if you have any other questions :)