Keda ScaledJob kafka-trigger scaling incorrectly

Question

Keda configuration is as below

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: job-message-consumer
spec:
  jobTargetRef:
    parallelism: 1
    template:
      spec:
        containers:
          - name: {{ .Values.JobMessageConsumer.name }}
            image: {{ .Values.image.repository }}
            args:
              - python
              - manage.py
              - jobs_consumer
            imagePullPolicy: {{ .Values.image.pullPolicy }}
            env:
              - name: SSL_CA_LOCATION
                value: {{ .Values.kafka.sslCaDirMount }}/ca.crt
              - name: SSL_CA_DIR_MOUNT
                value: {{ .Values.kafka.sslCaDirMount }}
              - name: SSL_SECRET_NAME
                value: {{ .Values.kafka.sslSecretName }}
              - name: SERVICE_CONSUMER_GROUP_ID
                value: {{ .Values.kafka.CONSUMER_GROUP_ID }}
              - name: KAFKA_SERVICE_JOBS_TOPIC
                value: {{ .Values.job_consumer_service.consumption_topic }}
              {{- include "env" . | nindent 14 }}
            resources:
              limits:
                cpu: "2"
                memory: "4Gi"
              requests:
                cpu: "500m"
                memory: "500Mi"
            volumeMounts:
              - name: user-certs
                mountPath: {{ .Values.kafka.sslUserCertDir }}
              - name: ca-certs
                mountPath: {{ .Values.kafka.sslCaDirMount }}
        volumes:
          - name: user-certs
            secret:
              secretName: {{ .Release.Name }}-kafka-user-certs
          - name: ca-certs
            secret:
              secretName: kafka-cluster-ca-cert
  pollingInterval: 20
  maxReplicaCount: 100
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: "kafka-kafka-bootstrap.kafka:9093"
        consumerGroup: "JobConsumer"
        topic: {{ .Values.job_consumer_service.consumption_topic }}
        lagThreshold: "1"
        offsetResetPolicy: latest
      authenticationRef:
        name: {{ .Release.Name }}-keda-trigger-auth-kafka-credential

The keda-jobs are required to run long time to complete the task hence the ScaledJob. But with the above ScaledJobs configuration the keda jobs are scaling every 20 sec . I want the keda jobs to scale only based on the messages on the kafka-trigger and not based on the pollinginterval. Can anyone help me on this

Answer 1

Please add this parameter to the kafka trigger in the triggers section:

        scaleToZeroOnInvalidOffset: "true"

If this scaleToZeroOnInvalidOffset parameter, added in versions 2.7.0 and above, is set to "true", the consumers for a partition will be scaled to zero when that partition does not have a valid offset [1].

[1] - https://keda.sh/docs/2.7/scalers/apache-kafka/