Liveness probe failing but the endpoint is accessible from different pods

1k views Asked by At

I'm trying to implement a simple liveness probe in my helm chart deployment template. Below is my liveness probe configuration. Spring boot /actuator/health endpoint is used as the health check endpoint.

containers:
        - name: {{ .Release.Name }}-container
          image: {{ .Values.container.image }}
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: 8080
            failureThreshold: 5
            periodSeconds: 10
            initialDelaySeconds: 30
            timeoutSeconds: 25

This is the error I'm encountering (Tried adding a large initialDelay and also tried adding a startupProbe. Both did not work)

Liveness probe failed: Get http://x.x.x.x:8080/actuator/health: dial tcp x.x.x.x:8080: connect: connection refused

However I'm able to get a 200 response from different pods via this endpoint which are in the same ec2 instance and also different ec2 instances.

$k exec -it pod/test sh
# curl http://x.x.x.x:8080/actuator/health  -I
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked
Content-Type: application/vnd.spring-boot.actuator.v3+json
correlation-id: x-x-x-x-x
Date: Fri, 09 Oct 2020 14:04:56 GMT

Without the liveness probe, the app is working fine and I can access all the endpoints via port 8080.

Tried setting up livenessprobe to an nginx image and it works fine (So ruling out network issues)

Containers:
  liveness:
    Container ID:   docker://0af63462845d6a2b44490308147c73277d22aff56f993ca7c065a495ff97fcfa
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:c628b67d21744fce822d22fdcc0389f6bd763daac23a6b77147d0712ea7102d0
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 29 Sep 2020 15:53:17 +0530
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/ delay=2s timeout=1s period=2s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-57smz (ro)
0

There are 0 answers