EKS Fargate amazon-cloudwatch-observability addon is not working

Question

I install EKS fargate and addons with terraform. Enabled IRSA. Created 3 fargate profile: external-secrets, kube-system, amazon-cloudwatch. amazon-cloudwatch-observability pod started successfully on amazon-cloudwatch namespace. Checked the logs. There is no error or warning. Checked the aws-auth cm. I can see my fargate profiles there.

Attached CloudWatchAgentServerPolicy and AWSXrayWriteOnlyAccess roles to iam roles of external-secrets iam role and amazon-cloudwatch iam roles like written in here.

fargate_profile_pod_execution_role_arn and iam_role_arn are the same for a fargate profile.For example: arn:aws:iam::{account_id}:role/external-secrets-{date_prefix}) .

Install external-secrets with helm: helm install external-secrets \ external-secrets/external-secrets \ -n external-secrets \ --create-namespace \ --set installCRDs=true \ --set webhook.port=9443 . Started pods without any warning or error.

I expected to see external-secrets logs but didnt. log group is not created automatically. Tail the logs of observability pod logs. There is no movement. No error no success. So here is my question. What the heck should i do else to make it work?

I tried this approach also and it worked btw. amazon-cloudwatch-observability and aws-observability is not related(There is no connection between them it seems)

Answer 1

You could deploy using a service account with something like the following in terraform if you're using oidc for your cluster

module "cw_metrics_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "~> 4.7.0"

  create_role = true

  role_name = "cw-observability-role"

  provider_url = var.eks_cluster_url

  role_policy_arns = [
    "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy",
    "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess",
  ]
  number_of_role_policy_arns = 2

  oidc_fully_qualified_subjects = [
    "system:serviceaccount:amazon-cloudwatch:cloudwatch-agent",
  ]

  oidc_fully_qualified_audiences = [
    "sts.amazonaws.com"
  ]
}

You can then pass the service account role arn to the add-on

resource "aws_eks_addon" "cw_metrics" {
  cluster_name  = var.eks_cluster_name
  addon_name    = "amazon-cloudwatch-observability"

  service_account_role_arn = module.cw_metrics_role.iam_role_arn
}