Using terraform yamldecode to access multi level element

Question

I have a yaml file (also used in a azure devops pipeline so needs to be in this format) which contains some settings I'd like to directly access from my terraform module.

The file looks something like:

variables:
  - name: tenantsList
    value: tenanta,tenantb
  - name: unitName
    value: canary

I'd like to have a module like this to access the settings but I can't see how to get to the bottom level:

locals {
  settings = yamldecode(file("../settings.yml"))
}

module "infra" {
  source = "../../../infra/terraform/"
  unitname = local.settings.variables.unitName
}

But the terraform plan errors with this:

Error: Unsupported attribute

  on canary.tf line 16, in module "infra":
  16:   unitname  = local.settings.variables.unitName
    |----------------
    | local.settings.variables is tuple with 2 elements

This value does not have any attributes.

Answer 1

It seems like the main reason this is difficult is because this YAML file is representing what is logically a single map but is physically represented as a YAML list of maps.

When reading data from a separate file like this, I like to write an explicit expression to normalize it and optionally transform it for more convenient use in the rest of the Terraform module. In this case, it seems like having variables as a map would be the most useful representation as a Terraform value, so we can write a transformation expression like this:

locals {
  raw_settings = yamldecode(file("${path.module}/../settings.yml"))
  settings = {
    variables = tomap({
      for v in local.raw_settings.variables : v.name => v.value
    })
  }
}

The above uses a for expression to project the list of maps into a single map using the name values as the keys.

With the list of maps converted to a single map, you can then access it the way you originally tried:

module "infra" {
  source = "../../../infra/terraform/"
  unitname = local.settings.variables.unitName
}

If you were to output the transformed value of local.settings as YAML, it would look something like this, which is why accessing the map elements directly is now possible:

variables:
  tenantsList: tenanta,tenantb
  unitName: canary

This will work only if all of the name strings in your input are unique, because otherwise there would not be a unique map key for each element.

---

(Writing a normalization expression like this also doubles as some implicit validation for the shape of that YAML file: if variables were not a list or if the values were not all of the same type then Terraform would raise a type error evaluating that expression. Even if no transformation is required, I like to write out this sort of expression anyway because it serves as some documentation for what shape the YAML file is expected to have, rather than having to study all of the references to it throughout the rest of the configuration.)

Answer 2

With my multidecoder for YAML and JSON you are able to access multiple YAML and/or JSON files with their relative paths in one step.

Documentations can be found here:

Terraform Registry -

https://registry.terraform.io/modules/levmel/yaml_json/multidecoder/latest?tab=inputs

GitHub:

https://github.com/levmel/terraform-multidecoder-yaml_json

Usage

Place this module in the location where you need to access multiple different YAML and/or JSON files (different paths possible) and pass your path/-s in the parameter filepaths which takes a set of strings of the relative paths of YAML and/or JSON files as an argument. You can change the module name if you want!

module "yaml_json_decoder" {
  source  = "levmel/yaml_json/multidecoder"
  version = "0.2.1"
  filepaths = ["routes/nsg_rules.yml", "failover/cosmosdb.json", "network/private_endpoints/*.yaml", "network/private_links/config_file.yml", "network/private_endpoints/*.yml", "pipeline/config/*.json"]
}

Patterns to access YAML and/or JSON files from relative paths:

To be able to access all YAML and/or JSON files in a folder entern your path as follows "folder/rest_of_folders/*.yaml", "folder/rest_of_folders/*.yml" or "folder/rest_of_folders/*.json".

To be able to access a specific YAML and/or a JSON file in a folder structure use this "folder/rest_of_folders/name_of_yaml.yaml", "folder/rest_of_folders/name_of_yaml.yml" or "folder/rest_of_folders/name_of_yaml.json"

If you like to select all YAML and/or JSON files within a folder, then you should use "*.yml", "*.yaml", "*.json" format notation. (see above in the USAGE section)

YAML delimiter support is available from version 0.1.0!

WARNING: Only the relative path must be specified. The path.root (it is included in the module by default) should not be passed, but everything after it.

Access YAML and JSON entries

Now you can access all entries within all the YAML and/or JSON files you've selected like that: "module.yaml_json_decoder.files.[name of your YAML or JSON file].entry". If the name of your YAML or JSON file is "name_of_your_config_file" then access it as follows "module.yaml_json_decoder.files.name_of_your_config_file.entry".

Example of multi YAML and JSON file accesses from different paths (directories)

first YAML file:

routes/nsg_rules.yml

rdp:
  name: rdp
  priority: 80
  direction: Inbound
  access: Allow
  protocol: Tcp
  source_port_range: "*"
  destination_port_range: 3399
  source_address_prefix: VirtualNetwork
  destination_address_prefix: "*"
  
---
  
ssh:
  name: ssh
  priority: 70
  direction: Inbound
  access: Allow
  protocol: Tcp
  source_port_range: "*"
  destination_port_range: 24
  source_address_prefix: VirtualNetwork
  destination_address_prefix: "*"

second YAML file:

services/logging/monitoring.yml

application_insights:
  application_type: other
  retention_in_days: 30
  daily_data_cap_in_gb: 20
  daily_data_cap_notifications_disabled: true
  logs:
  # Optional fields
   - "AppMetrics"
   - "AppAvailabilityResults"
   - "AppEvents"
   - "AppDependencies"
   - "AppBrowserTimings"
   - "AppExceptions"
   - "AppExceptions"
   - "AppPerformanceCounters"
   - "AppRequests"
   - "AppSystemEvents"
   - "AppTraces"

first JSON file:

test/config/json_history.json

{
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}

main.tf

module "yaml_json_multidecoder" {
  source  = "levmel/yaml_json/multidecoder"
  version = "0.2.1"
  filepaths = ["routes/nsg_rules.yml", "services/logging/monitoring.yml", test/config/*.json]
}

output "nsg_rules_entry" {
  value = module.yaml_json_multidecoder.files.nsg_rules.aks.ssh.source_address_prefix
}

output "application_insights_entry" {
  value = module.yaml_json_multidecoder.files.monitoring.application_insights.daily_data_cap_in_gb
}

output "json_history" {
  value = module.yaml_json_multidecoder.files.json_history.glossary.title
}

---

Changes to Outputs:

• nsg_rules_entry = "VirtualNetwork"
• application_insights_entry = 20
• json_history = "example glossary"