Unable to connect internet/google.com from pod. Docker and k8 are able to pull images

Question

I am trying to learn Kubernetes.

Create a single-node Kubernetes Cluster on Oracle Cloud using these steps here

cat /etc/resolv.conf
>> nameserver 169.254.169.254

kubectl run busybox --rm -it --image=busybox --restart=Never -- sh
cat /etc/resolv.conf
>> nameserver 10.33.0.10

nslookup google.com
>>Server:         10.33.0.10
Address:        10.33.0.10:53

;; connection timed out; no servers could be reached

ping 10.33.0.10
>>PING 10.33.0.10 (10.33.0.10): 56 data bytes

kubectl get svc  -n kube-system -o wide
>> CLUSTER-IP - 10.33.0.10

kubectl logs --namespace=kube-system -l k8s-app=kube-dns
>>[ERROR] plugin/errors: 2 google.com. A: read udp 10.32.0.9:57385->169.254.169.254:53: i/o timeout

Not able to identify if this is an error of coredns or pod networking. Any direction would really help

Answer 1

• Kubernetes has deprecated Docker as a container runtime after v1.20.
• Kubernetes Development decision to deprecate Docker as an underlying runtime in favor of runtimes that use the Container Runtime Interface (CRI) created for Kubernetes.
• To support this Mirantis and Docker came to the rescue by agreeing to partner in the maintenance of the shim code standalone.

More details here here

sudo systemctl enable docker
# -- Installin cri-dockerd
VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d '"' -f 4)
echo $VER
wget https://github.com/Mirantis/cri-dockerd/releases/download/${VER}/cri-dockerd-${VER}-linux-arm64.tar.gz
tar xvf cri-dockerd-${VER}-linux-arm64.tar.gz
install -o root -g root -m 0755 cri-dockerd /usr/bin/cri-dockerd
cp cri-dockerd /usr/bin/
# -- Verification
cri-dockerd --version
# -- Configure systemd units for cri-dockerd
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo cp cri-docker.socket cri-docker.service /etc/systemd/system/ 
sudo cp cri-docker.socket cri-docker.service /usr/lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket
# -- Using cri-dockerd on new Kubernetes cluster
systemctl status docker | grep Active

Answer 2

I ran into similar issue with almost same scenario described above. The accepted solution https://stackoverflow.com/a/72104194/1119570 is wrong. This issue is a pure networking issue that is not related to any of EKS upgrade in any way. The root cause for our issue was the fact that the Worker Node AWS EKS Linux 1.21 AMI being hardened by our security department which turns off the following setting in this file /etc/sysctl.conf:

net.ipv4.ip_forward = 0 After switching this setting to: net.ipv4.ip_forward = 1 and rebooting the EC2 Node, everything started working properly. Hope this helps!