data manipulation or enrichment in Cloud Workflows

Question

How to do simple data manipulation or enrichment?

Usually, in a pipeline, you do with DataFlow job, but are there options in Cloud Workflows to do that sort of thing?

Answer 1

You can achieve this functionality by declaring a Dictionary and reusing to collect things on the go.

The caveat is that you need to retype all the keys again when you add a new value, as variable reference, merge keys, or YAML anchors are not yet implemented.

So if you start with:

- initDictionary:
    assign:
      - myDictionary:
          name: John
          lastName: Smith
          dayOfTheWeek: ${currentTime.body.dayOfTheWeek}

you could later add a key from another result, like from Wikipedia like this:

- enrichStep:
    assign:
      - myDictionary:
          name: ${myDictionary.name}
          lastName: ${myDictionary.lastName}
          dayOfTheWeek: ${currentTime.body.dayOfTheWeek}
          wikiResult: ${wikiResult.body[1]}

here is a full wikipedia_enrichment.yaml sequence for you.

- getCurrentTime:
    call: http.get
    args:
        url: https://us-central1-workflowsample.cloudfunctions.net/datetime
    result: currentTime
- initDictionary:
    assign:
      - myDictionary:
          name: John
          lastName: Smith
          dayOfTheWeek: ${currentTime.body.dayOfTheWeek}
- readWikipedia:
    call: http.get
    args:
        url: https://en.wikipedia.org/w/api.php
        query:
            action: opensearch
            search: ${currentTime.body.dayOfTheWeek}
    result: wikiResult
- enrichStep:
    assign:
      - myDictionary:
          name: ${myDictionary.name}
          lastName: ${myDictionary.lastName}
          dayOfTheWeek: ${currentTime.body.dayOfTheWeek}
          wikiResult: ${wikiResult.body[1]}
- returnResult:
    return: ${myDictionary}