KEDA kafka - azure eventhub configuration

Question

I am having trouble when configuring ScaledObject for a kafka cluster although I've read de documentation I can't seem to understand why this isn't working I think I'm misconfiguring the appropiate scaled object.

I have configured a kafka trigger but this hasn't been working so instead I had tried to use a azure-eventhub one but don't know if this is the correct approach.

Having an issue when trying to configure a keda kafka scaled object with my triggerAuthentication I was trying to configure a plaintext authentication on kafka using the eventhub connection string as password like this:

ScaledObject.yaml

YAML

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scaled-object-name
spec:
  scaleTargetRef:
    name: app-name
  pollingInterval: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: <bootstrap-servers>
        consumerGroup: <consumer-group>     # This consumer group name has to be the one is consuming topics
        topic: <topic-name>
        tls: disable
        sasl: plaintext
        # Optional
        lagThreshold: "50"
        offsetResetPolicy: latest
      authenticationRef:
        name: keda-trigger-auth-kafka-credential

kafka-keda-secret

YAML

apiVersion: v1
kind: Secret
metadata:
  name: kafka-auth-secret
type: Opaque
stringData:
  username: "$ConnectionString"
  password: "Endpoint=sb://{namespace}.servicebus.windows.net/;SharedAccessKeyName={policy-name};SharedAccessKey={policy-key}"

TriggerAuthentication

YAML

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
spec:
  secretTargetRef:
    - parameter: username
      name: keda-kafka-secrets
      key: username
    - parameter: password
      name: keda-kafka-secrets
      key: password
 

But I wasn't able to it throws this error

Events:
  Type     Reason                   Age                 From           Message
  ----     ------                   ----                ----           -------
  Warning  KEDAScalerFailed         14m (x99 over 22h)  keda-operator  error parsing kafka metadata: no username given
  Warning  ScaledObjectCheckFailed  14m (x99 over 22h)  keda-operator  failed to ensure HPA is correctly created for ScaledObject

So instead I'm trying to configure an azure-eventhub ScaledObject like this

ScaledObjectAZEH

YAML
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scaled-object-az-eh
spec:
  scaleTargetRef:
    name: app-name
  pollingInterval: 30
  triggers:
    - type: azure-eventhub
      metadata:
        eventHubInfo:
          connectionStringFromEnv: EVENTHUB_CONNECTION_STRING
        consumerGroup: $Default
        partitionCount: 2
        storageConnectionFromEnv: ""

Do you have any recomendation on how to configure KEDA on kafka or azure eventhub?

Any help would be appreciate.

Answer 1

I have solved my issue by finding a misconfiguration on my ScaledObject

ScaledObject.yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scaled-object-name
spec:
  scaleTargetRef:
    name: app-name
  pollingInterval: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: <bootstrap-servers>
        consumerGroup: <consumer-group>     # This consumer group name has to be the one is consuming topics
        topic: <topic-name>
        # Optional
        lagThreshold: "50"
        offsetResetPolicy: latest
      authenticationRef:
        name: keda-trigger-auth-kafka-credential

TriggerAuthentication.yaml

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
spec:
  secretTargetRef:
    - parameter: username
      name: keda-kafka-secrets
      key: username
    - parameter: password
      name: keda-kafka-secrets
      key: password
    - parameter: sasl
      name: keda-kafka-secrets
      key: sasl
    - parameter: tls
      name: keda-kafka-secrets
      key: tls

Secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: kafka-auth-secret
type: Opaque
stringData:
  tls: "enable"
  sasl: "plaintext"
  username: <username>
  password: <azure-evnthub-connection-string>