I'm currently investigating the usage of AWS NAT Gateway to optimize costs for my EKS cluster. According to CloudWatch metrics, it appears that there's an approximately 80MB data transfer through the NAT Gateway each time a node from my managed node group is initiated in a private subnet. These traffic spikes, associated with node startup, are easily identifiable since there is no additional traffic during node lifetime.
I've already added VPC endpoints to my VPC for ECR (both DRK and API) and S3 services. Before adding them, the traffic amount was higher due to image pulling from ECR and S3, thus it should not be something related to image pulling.
I can't understand why there is this traffic on node launch. Any insight?
A few details:
- EKS version 1.23
- AMI version 1.23.17-20231116
EDIT: by analyzing flow logs, I discovered that the traffic is directed towards a pair of AWS CloudFront instances. I currently don't have one in my architecture, thus it should be something related to the AMI pulling process during node initialization.