How to make k8s Pod (generated by Jenkins) use Service account IAM role to access AWS resources

Question

Follow this link, I can create a pod whose service account's role can access the AWS resources; so the pod can access them either.

Then, inspired by this EKS-Jenkins-Workshop, I change this workshop a little bit. I want to deploy Jenkins Pipeline, this Jenkins Pipeline can create a pod whose account service's role can access aws resources, but the problem is the cdk code in this pod cannot access AWS resources. (I write the cdk code to access AWS resources, reference (Your first AWS CDK app)[https://docs.aws.amazon.com/cdk/latest/guide/hello_world.html])

This is my Jenkinsfile

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
metadata:
  name: jenkins-agent
  Namespace: default
spec:
  serviceAccountName: jenkins
  containers:
  - name: node-yuvein
    image: node
    command:
    - cat
    tty: true
"""
    }
  }

  stages {
    stage('Build') {
      steps {
        container('node-yuvein') {
          dir('hello-cdk'){
            sh "pwd"
            sh 'npm --version'
            sh 'node -v'
            sh 'npm install -g typescript'
            sh 'npm install -g aws-cdk'
            sh 'npm install @aws-cdk/aws-s3'
            sh 'npm run build'
            sh 'cdk deploy'
          }
        }
      }
    } 
  }
}

When I run the pipeline, it has this error:

User: arn:aws:sts::450261875116:assumed-role/eksctl-eksworkshop-eksctl3-nodegr-NodeInstanceRole-1TCVDYSM1QKSO/i-0a4df3778517df0c6 is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-west-2:450261875116:stack/HelloCdkStack/*

I am a beginner of K8s, Jenkins and cdk. Hope someone can help me. Thanks a lot.

Further Debugging:

1. In Jenkins Console, I can get serviceAccountName: "jenkins", and the name of my service account in EKS is jenkins.
2. the pod also get correct ENV:

+ echo $AWS_ROLE_ARN
arn:aws:iam::450261875116:role/eksctl-eksworkshop-eksctl3-addon-iamservicea-Role1-YYYFXFS0J4M2

+ echo $AWS_WEB_IDENTITY_TOKEN_FILE
/var/run/secrets/eks.amazonaws.com/serviceaccount/token

3. The node.js and npm I installed are the lastest version.

+ npm --version
6.14.8

+ node -v
v14.13.0

+ aws sts get-caller-identity
{
    "UserId": "AROAWRVNS7GWO5C7QJGRF:botocore-session-1601436882",
    "Account": "450261875116",
    "Arn": "arn:aws:sts::450261875116:assumed-role/eksctl-eksworkshop-eksctl3-addon-iamservicea-Role1-YYYFXFS0J4M2/botocore-session-1601436882"
}

when I run this command, it appears my service account role. But I still get the original error.

Answer 1

Jenkins podTemplate has serviceAccount option: https://github.com/jenkinsci/kubernetes-plugin#pod-and-container-template-configuration

• Create an IAM role mapped to an EKS cluster
• Create a ServiceAccount mapped to an IAM role
• Pass ServiceAccount name to a podTemplate

Further debugging:

1. Ensure the pod has correct service account name.
2. Check if pod got AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE env vars (they are added automatically).
3. Check if AWS SDK you use is above the minimal version: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
4. Run aws sts get-caller-identity to see the role, don't waste time on running an actual job.

Answer 2

In the case of working with Jenkins slaves, one needs to customize the container images to use AWS CLI V2 instead of AWS CLI V1. I was running into errors related to authorization like the question poses; my client was using the cluster node roles instead of using the assumed web identity role of my service account attached to my Jenkins-pods for the slave containers.

Apparently V2 of the AWS CLI includes the web identity token file as part of the default credentials chain whereas V1 does not.

Here's a sample Dockerfile that pulls the latest AWS CLI version so this pattern works.

FROM jenkins/inbound-agent

# run updates as root
USER root

# Create docker group
RUN addgroup docker

# Update & Upgrade OS
RUN apt-get update
RUN apt-get -y upgrade

#install python3
RUN apt-get -y install python3


# add AWS Cli version 2 for web_identity_token files
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install

# Add Maven
RUN apt-get -y install maven --no-install-recommends

# Add docker
RUN curl -sSL https://get.docker.com/ | sh
RUN usermod -aG docker jenkins

# Add docker compose
RUN curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
RUN chmod +x /usr/local/bin/docker-compose


# Delete cached files we don't need anymore:
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/*

# close root access
USER jenkins

Further, I had to make sure my serviceaccount was created and attached to both the Jenkins master image and the jenkins slaves. This can be accomplished via Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds -> Pod Template Details.

Be sure to edit Namespace and Serviceaccount fields with the appropriate values.