Error running Canary Deployment in Spinnaker

Question

I am trying to enable the canary deployment for the AWS eks but my kayenta pod is not starting. When I describe the pod I see this error. Can anyone help?

  Warning  Unhealthy  12m (x2 over 12m)     kubelet            Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused
  Warning  Unhealthy  2m56s (x59 over 12m)  kubelet            Readiness probe failed: wget: server returned error: HTTP/1.1 503

This is the status of pod:

NAME                               READY   STATUS    RESTARTS   AGE
spin-clouddriver-d796bdc59-tpznw   1/1     Running   0          3h40m
spin-deck-77cc75b57d-w7rfp         1/1     Running   0          3h40m
spin-echo-db954bb9-phfd5           1/1     Running   0          3h40m
spin-front50-7c5684cf9-t7vl8       1/1     Running   0          3h40m
spin-gate-78d6779854-7xqz4         1/1     Running   0          3h40m
spin-kayenta-6d7b5fdfc6-p5tcp      0/1     Running   0          21m
spin-kayenta-869c46bfcf-8t5fh      0/1     Running   0          28m
spin-orca-7ddd66758d-mpnkg         1/1     Running   0          3h40m
spin-redis-5975cfcdc8-rnm9g        1/1     Running   0          45h
spin-rosco-b7dbb577-z4szz          1/1     Running   0          3h40m

Answer 1

I will try to address your issue from the Kubernetes perspective.

The errors you were experiencing:

  Warning  Unhealthy  12m (x2 over 12m)     kubelet            Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused
  Warning  Unhealthy  2m56s (x59 over 12m)  kubelet            Readiness probe failed: wget: server returned error: HTTP/1.1 503

indicates that there was a problem with your ReadinessProbe configuration. Removing the ReadinessProbe from the deployment "fixed" the error but can cause some more issues in the future. To avoid that I recommend adding it back with a proper configuration:

Probes have a number of fields that you can use to more precisely control the behavior of liveness and readiness checks:

• initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.

• periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.

• timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.

• successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. Minimum value is 1.

• failureThreshold: When a probe fails, Kubernetes will try failureThreshold times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1.

You'll need to adjust the Probe's configuration based on your apps behavior (usually by trial and error). The two resources I would recommend that will help you with that are:

• Configure Liveness, Readiness and Startup Probes

• Kubernetes best practices: Setting up health checks with readiness and liveness probes