How to setup Kibana (EFK stack) on Azure Kubernetes cluster using reverse proxy ingress?

Question

Firstly wants to thank you for an amazing article above.

I have a scenario I want to get some help with, So I have installed Elastic search, Filebeat and Kibana on AKS cluster. Now, since I am using Nginx Ingress controller to expose application on Reverse proxy load balancer which is hooked to a hostname say http://example.com

I am not able to expose kibana to outside. I get 404 error. I have tried to add serverbasepath variable in kibana deployment etc. but still get 404 error.

• name: SERVER_BASEPATH value: "/kibana"

Below is my setup: 1.Ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-dev
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: frontend-ui-service
          servicePort: 80
        path: /(.*)
      - backend:
          serviceName: home-micro-service
          servicePort: 3333
        path: /api-dev(/|$)(.*)
      - backend:
          serviceName: kibana-kibana
          servicePort: 5601
        path: /kibana(/|$)(.*)

2.Kibana - values.yaml

---
elasticsearchHosts: "http://elasticsearch-master:9200"

replicas: 1

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs:
#  - name: "NODE_OPTIONS"
#    value: "--max-old-space-size=1800"
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# Allows you to load environment variables from kubernetes secret or config map
envFrom: []
# - secretRef:
#     name: env-secret
# - configMapRef:
#     name: config-map

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: kibana-keystore
#    secretName: kibana-keystore
#    path: /usr/share/kibana/data/kibana.keystore
#    subPath: kibana.keystore # optional

image: "dockerRepo/docker.elastic.co/kibana/kibana"
imageTag: "7.9.1"
imagePullPolicy: "IfNotPresent"

# additionals labels
labels: {}

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

resources:
  requests:
    cpu: "1000m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

protocol: http

serverHost: "0.0.0.0"

healthCheckPath: "/app/kibana"

# Allows you to add any config files in /usr/share/kibana/config/
# such as kibana.yml
kibanaConfig: {}
#   kibana.yml: |
#     key:
#       nestedkey: value

# If Pod Security Policy in use it may be required to specify security context as well as service account

podSecurityContext:
  fsGroup: 1000

securityContext:
  capabilities:
    drop:
    - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

serviceAccount: ""

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

httpPort: 5601

extraContainers: ""
# - name: dummy-init
#   image: busybox
#   command: ['echo', 'hey']

extraInitContainers: ""
# - name: dummy-init
#   image: busybox
#   command: ['echo', 'hey']

updateStrategy:
  type: "Recreate"

service:
  type: ClusterIP
  loadBalancerIP: ""
  port: 5601
  nodePort: ""
  labels: {}
  annotations: {}
    # cloud.google.com/load-balancer-type: "Internal"
    # service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    # service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    # service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
    # service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
  loadBalancerSourceRanges: []
    # 0.0.0.0/0

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

imagePullSecrets: []
nodeSelector: {}
tolerations: []
affinity: {}

nameOverride: ""
fullnameOverride: ""

lifecycle: {}
  # preStop:
  #   exec:
  #     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
  # postStart:
  #   exec:
  #     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]

# Deprecated - use only with versions < 6.6
elasticsearchURL: "" # "http://elasticsearch-master:9200"

Answer 1

Your image version is 7.9.1, as mentioned here

server.rewriteBasePath:

Specifies whether Kibana should rewrite requests that are prefixed with server.basePath or require that they are rewritten by your reverse proxy. In Kibana 6.3 and earlier, the default is false. In Kibana 7.x, the setting is deprecated. In Kibana 8.0 and later, the default is true. Default: deprecated

As far as I know if you want to use server.basePath then additionally you have to set server.rewriteBasePath: true

Try to add the following to the Kibana config:

server.basePath: "/kibana"
server.rewriteBasePath: true

Additionally, as mentioned here by @anyasabo

You would need to override the path for the readiness probe in the podtemplate of your kibana resource. Currently it is hardcoded to /login

---

If that won't work I would suggest to check if ingress is configured properly.

For example create simple ingress with just / path to kibana and check if it works.

---

Additional resources:

• Kibana deployed to kubernetes cluster returns 404