aws-iam-authenticator returns already expired token - eks

Question

We are running an EKS cluster with nodes created with a node group based on AWS autoscaling group.

The cluster is created using eksctl.

The node works ok and after a while, it fails to communicate to the Kubernetes API.

Digging into the problem we found the following. The aws-iam-authenticator returns tokens that fail when used to access the API.

Finally, we found the following situation, we tried to verify the token using the same aws-iam-authenticator, but the validations fails with the message that the token is expired.

We check the situation running the following command:

aws-iam-authenticator verify -i preguntados-dev-v1 -t $( aws-iam-authenticator token -i preguntados-dev-v1 | jq -r '.status.token' )

It requests a token, extracts the token value from the returned JSON, and sends it to verify.

The response is (in the problematic node):

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403). Body: {"Error":{"Code":"ExpiredToken","Message":"The security token included in the request is expired","Type":"Sender"},"RequestId":"477f5292-0e87-4734-bd60-74720a3e13a8"}

aws-iam-authenticator version
{"Version":"v0.5.0","Commit":"1cfe2a90f68381eacd7b6dcfa2bf689e76eb8b4b"}

Answer 1

I have faced a similar issue and it is mainly because of reasons regarding system time being pulled off. If your Local time is incorrect then correcting it will be a key or else, run this command on Linux(Ubuntu):

sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"

Source: https://askubuntu.com/a/655528