How to merge unique values for a child array when merging objects

Question

I want to merge the .environments.default and .environments.dev objects together, so that dev values override any default values and default values fill in anything not defined in the dev object. If I use .environments.default * .environments.dev most of the objects are merged correctly, except for the array lists under the env and secrets objects. The entire dev object is used, but I want the array lists to be merged with unique values.

Is there anyway to get this result with jq? If it helps I could also try doing this in yq with the YAML file.

I would also really like to know if this is possible without hardcoding "env" or "secrets" anywhere in the code.

When I use .environments.default * .environments.dev on the below source...

Source:

{
"environments": {
    "default": {
      "config": {
        "security": {
          "user": 0,
          "group": 0
        },
        "resources": {
          "limits": {
            "memory": "256Mi",
            "cpu": "500m"
          }
        },
        "env": [
          {
            "name": "DEFAULT_ENV",
            "value": "DEFAULT_ENV_VALUE"
          },
          {
            "name": "BASE_URL",
            "value": "DEFAULT_BASE_URL"
          }
        ],
        "secrets": [
          {
            "secretID": "567",
            "mountPath": "/default/dir/to/567/path"
          },
          {
            "secretID": "123",
            "mountPath": "/default/dir/to/123/path"
          },
          {
            "secretUUID": "789",
            "mountPath": "/default/dir/to/789/path"
          }
        ]
      }
    },
    "dev": {
      "config": {
        "replicas": 1,
        "resources": {
          "limits": {
            "cpu": "5000m"
          }
        },
        "env": [
          {
            "name": "BASE_URL",
            "value": "DEV_BASE_URL"
          },
          {
            "name": "DEV_ENV_1",
            "value": "DEV_ENV_1"
          }
        ],
        "secrets": [
          {
            "secretID": "456",
            "mountPath": "/dev/dir/to/456/path"
          },
          {
            "secretID": "123",
            "mountPath": "/dev/dir/to/456/path"
          }
        ]
      }
    }
  }
}

I get...

Current output:

{
  "config": {
    "security": {
      "user": 0,
      "group": 0
    },
    "resources": {
      "limits": {
        "memory": "256Mi",
        "cpu": "5000m"
      }
    },
    "env": [
      {
        "name": "BASE_URL",
        "value": "DEV_BASE_URL"
      },
      {
        "name": "DEV_ENV_1",
        "value": "DEV_ENV_1"
      }
    ],
    "secrets": [
      {
        "secretID": "456",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "123",
        "mountPath": "/dev/dir/to/456/path"
      }
    ],
    "replicas": 1
  }
}

but I want...

Desired output:

{
  "config": {
    "security": {
      "user": 0,
      "group": 0
    },
    "resources": {
      "limits": {
        "memory": "256Mi",
        "cpu": "5000m"
      }
    },
    "env": [
      {
        "name": "BASE_URL",
        "value": "DEV_BASE_URL"
      },
      {
        "name": "DEV_ENV_1",
        "value": "DEV_ENV_1"
      },
      {
        "name": "DEFAULT_ENV",
        "value": "DEFAULT_ENV_VALUE"
      }
    ],
    "secrets": [
      {
        "secretID": "456",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "123",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "567",
        "mountPath": "/default/dir/to/567/path"
      },
      {
        "secretUUID": "789",
        "mountPath": "/default/dir/to/789/path"
      }
    ],
    "replicas": 1
  }
}

Answer 1

From the comments thread:

The env list should be unique by name and the secrets list should be unique by secretID. […] I was wondering if it could be unique by the first object key in the array? […] name always comes before value and secretID always comes before mountPath

Object fields don't have an order conveyed by the format, only array items do. The order you observe in a specific file is just its current representation.

Just for the fun of it, how would you do it anyway?

So, let the fun begin:

This captures and iterates over all items of non-empty arrays from the .default branch (saving the array path with paths(arrays | select(has(0))), each item's value with getpath(…)[], and the first item's key and value using to_entries[0]). The iteration's base document is the merge you already performed, but with these arrays removed from the .default branch, so the saved items can be individually tested and appended to it if they pass the condition that no other first key's value matches theirs: IN(.[][$v0.key]; $v0.value) | not.

.environments | [.default | paths(arrays | select(has(0))) as $p
  | getpath($p)[] as $v | {$p,$v,v0:($v | to_entries[0])}
] as $a | reduce $a[] as {$p,$v,$v0} ((.default | delpaths([$a[].p])) * .dev;
  setpath($p; getpath($p) | . + [select(IN(.[][$v0.key]; $v0.value) | not) | $v])
)

Demo