Python YAML parameter reference (get value from other parameter)

Question

I'm trying to use a YAML config file in my Python script, one of the problems I've found is that I can't access other attributes, so I need to duplicate a lot of content.

For example

root_path: /root
script_path: root_path + /scripts

This doesn't exists of course, but is there some way to achieve this? Because there are a lot of contents that I can't duplicate because when one changes, I need to change it everywhere...

I've also looked at creating my own join function

root_path: &ROOT /root
script_path: !join [*ROOT, '/scripts']


def join(loader, node):
    seq = loader.construct_sequence(node)
    return ''.join([str(i) for i in seq])

yaml.add_constructor('!join', join)

But I need to set a &VARNAME every time, it would be nice to automatically set the parameter key as the referenceable object...

Answer 1

Well, I've been working on a solution because nothing was what I wanted and everything was so complex I wanted to die...

This solutions transforms a %something% string into the value for something.

It works perfectly, this is an example

root_path: /root
script_path: "%root%/scripts"

With this method, script_path will become /root/scripts.

def replace_values(yaml_file):
    def _get(dict, list):
        return reduce(lambda d, k: d[k], list, dict)

    def _replace(obj):
        for k, v in obj.iteritems():
            if isinstance(v, dict):
                _replace(v)
            if isinstance(v, str):
                match = re.match(r'%(.*)%', v)
                if match:
                    reference = match.group(1).split('.')
                    replace = _get(yaml_file, reference)
                    obj[k] = re.sub(r'%.*%', replace, v)

    _replace(yaml_file)
    return yaml_file

Usage is easy, just load a Yaml file the normal way, and call replace.

with open(config_file.get(options.env), 'r') as ymlfile:
    config = yaml.load(ymlfile)

config = replace_values(config)

Then our new config will store the replaced values, it doesn't overwrite the original .yml file of course. I hope you find this useful, I really do because it was exactly what I needed.

I used a percentage % token, but you can change it to whatever you want and change the method to make it work with regex (some tokens are used by regex that's why I used %)

Answer 2

You can't really do that with YAML. For a config file, you could instead use ConfigParser, which lets you interpolate values so that your example would look like this:

root_path: /root
script_path: %(root_path)s/scripts 

Answer 3

I would just tweak the YAML parser a bit ¹:

import ruamel.yaml as yaml
from ruamel.yaml.comments import CommentedMap

yaml_str = """\
root_path: /root   # use this key for expansion of values
script_path: root_path + /scripts
"""

def set_item(self, key, value):
    split_val = value.split(' + ', 1)
    if len(split_val) > 1:
        alt_key = split_val[0].strip()
        if alt_key in self.keys():
            value = self.get(alt_key) + split_val[1]
    self._org__setitem__(key, value)

CommentedMap._org__setitem__ = CommentedMap.__setitem__
CommentedMap.__setitem__ = set_item

data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)
print yaml.dump(data, Dumper=yaml.RoundTripDumper)

will give you:

root_path: /root   # use this key for expansion of values
script_path: /root/scripts

Please note that both the comment and the ordering of the keys are preserved with the RoundTripLoader/Dumper combination. data works as a normal Python dict.

---

¹ _{This was done using ruamel.yaml of which I am the author. It is a superset of PyYAML primarily targeted at preserving comment and other information for data round-trips.}