yaml/ruamel string formatting and printing output isn't working as expected

Question

1. I have the following python dit string format as shown below.

   # With two dictionary input
   dit = "{p_d: '{a:3, what:3.6864e-05, s:lion, vec_mode:'{2.5, -2.9, 3.4, 5.6, -8.9, -5.67, 2, 2, 2, 2, 5.4, 2, 2, 6.545, 2, 2}, sst:'{c:-20, b:6, p:panther}}}\n{mgbp: '{ifftp:'{ipdp:'{ipdncf:'{r_t:no, mfn:\"bbbb.txt\", r_s:-967901775, np:-1187634210}}}}"
   
   # With single dictionary input
   #1 dit = "'{p_d: '{a:3, what:3.6864e-05, s:lion, vec_mode:'{2.5, -2.9, 3.4, 5.6, -8.9, -5.67, 2, 2, 2, 2, 5.4, 2, 2, 6.545, 2, 2}, sst:'{c:-20, b:6, p:panther}}}"
   

2. And I am trying to use the ruamel.yaml to dump it as yaml output file with the below code.

import sys
from pathlib import Path
import ruamel.yaml

path = Path('yamloutput.yaml')

#1 dit = "'{p_d: '{a:3, what:3.6864e-05, s:lion, vec_mode:'{2.5, -2.9, 3.4, 5.6, -8.9, -5.67, 2, 2, 2, 2, 5.4, 2, 2, 6.545, 2, 2}, sst:'{c:-20, b:6, p:panther}}}"

dit = "{p_d: '{a:3, what:3.6864e-05, s:lion, vec_mode:'{2.5, -2.9, 3.4, 5.6, -8.9, -5.67, 2, 2, 2, 2, 5.4, 2, 2, 6.545, 2, 2}, sst:'{c:-20, b:6, p:panther}}}\n{mgbp: '{ifftp:'{ipdp:'{ipdncf:'{r_t:no, mfn:\"bbbb.txt\", r_s:-967901775, np:-1187634210}}}}"


yaml_str = dit.replace('"', '').replace("'",'').replace(':', ': ').replace('{','[').replace('}',']')

def cleanup(d):
    if isinstance(d, list):
        ret_val = {}
        for elem in d:
            assert len(elem) == 1 and isinstance(elem, dict)
            ret_val.update(elem)
        return ret_val
    return d
    
yaml = ruamel.yaml.YAML(typ='safe')
yaml.default_flow_style = False
data = yaml.load(yaml_str)
data = cleanup(data)
data['p_d'] = cleanup(data['p_d'])
data['p_d']['sst'] = cleanup(data['p_d']['sst'])

yaml.dump(data, path)

sys.stdout.write(path.read_text())

3. When executed the above code I was getting the below error.

File "_ruamel_yaml.pyx", line 707, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.parser.ParserError: did not find expected <document start>
  in "<unicode string>", line 3, column 1

4. So I tried include the document start format using --- and inserted new line character \n between each dictionary in the string as shown below

dit = "---\n{p_d: '{a:3, what:3.6864e-05, s:lion, vec_mode:'{2.5, -2.9, 3.4, 5.6, -8.9, -5.67, 2, 2, 2, 2, 5.4, 2, 2, 6.545, 2, 2}, sst:'{c:-20, b:6, p:panther}}}\n{mgbp: '{ifftp:'{ipdp:'{ipdncf:'{r_t:no, mfn:\"bbbb.txt\", r_s:-967901775, np:-1187634210}}}}"

5. Still getting the same above error. But if its run with single dictionary string it works fine.

6. Expected Output:

p_d:
  a: 3
  s: lion
  sst:
    b: 6
    c: -20
    p: panther
  vec_mode:
  - 2.5
  - -2.9
  - 3.4
  - 5.6
  - -8.9
  - -5.67
  - 2
  - 2
  - 2
  - 2
  - 5.4
  - 2
  - 2
  - 6.545
  - 2
  - 2
  what: 3.6864e-05
mgbp:
  ifftp:
    ipdp:
      ipdncf:
        r_t: no
        mfn: bbbb.txt
        r_s: -967901775
        np: -967901775

Query:

1. If I am having multiple dict string to be converted into yaml output, do I need to specify any other "document start" format ? if so what is I am missing in the above code ?

Please provide your comments.

Answer 1

The error is rather clear in that the parser expects a document start at line 3 column 1. So inserting something before line 1 is not going to help at all. If you want to insert ---\n do that after the first newline in your string variable (and then use `yaml.load_all()' to load a multi-document).

But that is not going to help you much, as the value for the key p_d is a single quoted string ('{a:3, what:3.6864e-05, s:lion, vec_mode:') followed by more characters ( {2.5,...... ) that is an invalid YAML sccalar.

So this is not going to work either

dit = "{p_d: '{a:3, what:3.6864e-05, s:lion, vec_mode:'{2.5, -2.9, 3.4, 5.6, -8.9, -5.67, 2, 2, 2, 2, 5.4, 2, 2, 6.545, 2, 2}, sst:'{c:-20, b:6, p:panther}}}\n---\n{mgbp: '{ifftp:'{ipdp:'{ipdncf:'{r_t:no, mfn:\"bbbb.txt\", r_s:-967901775, np:-1187634210}}}}"

It is really unclear what you are trying to do and at your level of comprehension of YAML, you cannot expect to resolve this by asking questions on StackOverflow.

If your knowledge of builing data structures in Python is better, maybe it works that you build the data-structure that you want in Python and then dump it, that way you will get the single quotes (if you need them at all) at the right place (i.e. the beginning and end of a scalar) and the resulting YAML output should be easily converted to a Python string that then is (still) valid YAML input.