Terraform AWS GovCloud Error: configuring Terraform AWS Provider, api error InvalidClientTokenId: security token invalid

Question

Problem

I am trying to deploy some stuff to govcloud, and am having a heck of a time getting this to deploy. The code has successfully deployed to the commercial aws infrastructure without issue. But throws an error about the "InvalidClientTokenId"

Any advice is appreciated, I'm using SO to see if I'm just being an idiot (most likely) or if I need to maybe open an issue on GitHub.

What I've done

• verified my credentials (~/.aws/{credentials, config})
• I've tried deploying a super simple bit of TF code (data only, no resources) which succeeded, so my creds seem to be successful.
• I've tried checking my ENV vars (via printenv) and found no extraneous AWS/TF variables
• I've tried to audit my govcloud accesses, and found that I was using a Session Token ASIAxxxxxxxxx but can't seem to find that token anywhere (aside from aws sts get-session-token)
• I've tried running both terragrunt plan & terraform plan but both result in the same error.

Notes

• I am using Terragrunt to wrap Terraform here
• TF Version : v1.4.5
• TG Version : v0.45.2

The Error

I'm pretty sure I have a rogue session token saved somewhere somehow, but I can't find it. I haven't tried deleting my ~/.aws dir yet, but I only have the config & credentials files there

Terraform planned the following actions, but then encountered a problem:

...
...
...

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: configuring Terraform AWS Provider: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, api error InvalidClientTokenId: The security token included in the request is invalid.
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on provider.tf line 27, in provider "aws":
│   27: provider "aws" {
│ 
╵
ERRO[0003] Terraform invocation failed in /path/to/tf/repo/tf-modules 
ERRO[0003] 1 error occurred:
        * exit status 1

Provider.tf

# Generated by Terragrunt.
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.62.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.19.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.9.0"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.2.1"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.2.1"
    }
  }
}

provider "aws" {
  region                   = var.region
  # sts_region               = var.region
  profile                  = "govcloud"
  shared_config_files      = [pathexpand("~/.aws/config")]
  shared_credentials_files = [pathexpand("~/.aws/credentials")]
}

Terragrunt.hcl (Basically the main.tf)

locals {
  name  = "terragrunt"
  cloud = "aws" # Only Valid [aws]

  version = {
    tf_aws  = "4.62.0",     # DONT TOUCH
    tf_k8s  = "2.19.0",     # DONT TOUCH
    tf_helm = "2.9.0",      # DONT TOUCH
    tf_http = "3.2.1",      # DONT TOUCH
    tf_null = "3.2.1"       # DONT TOUCH
  }

  environment = {
    aws = "govcloud" # Only Valid [eastwest, govcloud]
  }

  region = {
    govcloud = "us-gov-east-1",
    eastwest = "us-east-1"
  }
}

# Indicate the input values to use for the variables of the module.
inputs = {
  k8s_ver = "1.25"

  cloud_provider    = local.cloud
  cloud_environment = local.environment[local.cloud]

  instance_type   = "t2.medium"
  ...
  ... (Misc node configs)
  ... 
  region    = local.region[local.environment[local.cloud]]

  name      = "${local.name}"
  domain    = "domain.tld"
  subnet    = "10.11.0.0"

  helm_charts = [
    {
      name = "cert-manager"
    }, 
    {
      name = "code-server"
    }
  ]

  default_tags = {keys=values}
}

# =======================================================
# - - - - - - - DO NOT EDIT BELOW THIS LINE - - - - - - - 
# =======================================================

terraform {
  extra_arguments "common_vars" {
    commands = get_terraform_commands_that_need_vars()

    arguments = [
      "-var-file=./${local.name}.tfvars"
    ]
    
    env_vars = { # Set this b/c its what made the small seperate plan work.
      AWS_PROFILE = "govcloud"
    }
  }
}

# Indicate what region to deploy the resources into
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
...
... <Provider.tf contents>
...
EOF
}

~/.aws/config

[default]
region=us-east-1
output=yaml

[profile govcloud]
region=us-gov-east-1
output=yaml

[profile eastwest]
region=us-east-1
output=yaml

~/.aws/credentials

[default]
aws_access_key_id=<eastwest_access_key>
aws_secret_access_key=<eastwest_secret_key>

[govcloud]
aws_access_key_id=<govcloud_access_key>
aws_secret_access_key=<govcloud_secret_key>

[eastwest]
aws_access_key_id=<eastwest_access_key>
aws_secret_access_key=<eastwest_secret_key>

Answer 1

1. Confirm their is not multiple REGIONS being set and causing conflicts. For example: ~/.aws/config and the var.REGION could be different and cause this error.