Kafka pods apparently going beyond their resource limits in kubernetes

121 views Asked by At

I am using kafka Strimzi operator. I have made a deployment with resources request and limits as shown below.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  labels:
    app: strimzi
spec:
  kafka:
    version: 3.5.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    resources:
      requests:
        memory: 16Gi
        cpu: "8"
      limits:
        memory: 16Gi
        cpu: "8"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.4"
    storage:
        type: persistent-claim
        size: 80Gi
        deleteClaim: false
        class: longhorn
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      requests:
        memory: 4Gi
        cpu: "2"
      limits:
        memory: 4Gi
        cpu: "2"
    storage:
      type: persistent-claim
      size: 15Gi
      deleteClaim: false
      class: longhorn
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

Having a look at prometheus/grafana, I am sseing the resource consumption limit per kafka pod is surpassed as shown in in the figures below

enter image description here

enter image description here

Can you please what i am misinterpreting/misunderstading? In fact, my understading is that the resource limit of a pod can not be surpassed in any way, and it will lead to throttling in case of CPU, and to an OOM for the memory case!!!

0

There are 0 answers