KEDA NATS scaler not activating deployment

96 views Asked by At

I am in the process of sampling KEDA for potential use in a project. I have set it up locally on a minikube K8s and I have created a ScaledObject using the below configuration:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: message-scaledobject
spec:
  pollingInterval: 3    # Optional. Default: 30 seconds
  cooldownPeriod: 10   # Optional. Default: 300 seconds
  minReplicaCount: 0   # Optional. Default: 0
  maxReplicaCount: 5  # Optional. Default: 100
  scaleTargetRef:
    name: message
  triggers:
    - type: stan
      metadata:
        natsServerMonitoringEndpoint: "nats-headless.keda-test.svc.cluster.local:8222"
        queueGroup: "MESSAGE-QUEUE"
        durableName: "MESSAGE-SCALER"
        subject: "MESSAGE.inbound"
        lagThreshold: "1"
        activationLagThreshold: "2"
        useHttps: "false"

As you can see my setup is quite minimal and straightforward. I have a deployment that effectively acts as the target (a simple Spring boot app), a NATS cluster running with the same cluster (different namespace) and that's about it. Note that my NATS deployment has a single stream MESSAGE and two subjects:

  • MESSAGE.inbound
  • MESSAGE.outbound.

For the purposes of this demo, I am pushing messages to the MESSAGE.inbound subject.

Now, when I describe the ScaledObject I see that the configuration is correct and everything seems to be running OK. The output of the describe command is the one below:

Name:         message-scaledobject
Namespace:    keda-test
Labels:       app.kubernetes.io/managed-by=Helm
              scaledobject.keda.sh/name=message-scaledobject
Annotations:  meta.helm.sh/release-name: message
              meta.helm.sh/release-namespace: keda-test
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2023-12-02T13:57:24Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  23699
  UID:               6d069861-d66c-4ab3-8380-0683c8c31899
Spec:
  Cooldown Period:    10
  Max Replica Count:  5
  Min Replica Count:  0
  Polling Interval:   3
  Scale Target Ref:
    Name:  message
  Triggers:
    Metadata:
      Activation Lag Threshold:         2
      Durable Name:                     MESSAGE-SCALER
      Lag Threshold:                    1
      Nats Server Monitoring Endpoint:  nats-headless.keda-test.svc.cluster.local:8222
      Queue Group:                      MESSAGE-QUEUE
      Subject:                          MESSAGE.inbound
      Use Https:                        false
    Type:                               stan
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is not performed because triggers are not active
    Reason:   ScalerNotActive
    Status:   False
    Type:     Active
    Status:   Unknown
    Type:     Fallback
    Status:   Unknown
    Type:     Paused
  External Metric Names:
    s0-stan-MESSAGE-inbound
  Hpa Name:                keda-hpa-message-scaledobject
  Original Replica Count:  1
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type    Reason                      Age                    From           Message
  ----    ------                      ----                   ----           -------
  Normal  KEDAScalersStarted          8m36s                  keda-operator  Scaler stan is built.
  Normal  KEDAScalersStarted          8m36s                  keda-operator  Started scalers watch
  Normal  ScaledObjectReady           8m36s (x2 over 8m36s)  keda-operator  ScaledObject is ready for scaling
  Normal  KEDAScaleTargetDeactivated  8m36s                  keda-operator  Deactivated apps/v1.Deployment keda-test/message from 1 to 0

With all that in place I have a small script that bombards the NATS cluster with messages at this particular subject. I run it and I indeed verify that the messages are being received correctly. However the scaler never activates, leaving the deployment with 0 pods running.

I have tried both the NATS Streaming and NATS JetStream scalers and I can't get any of those run the activation phase. Note that in the case of the former, the app is scaling correctly based on the rules defined when the minReplicaCount is 1 which means that the deployed HPA is working correctly and KEDA serves metrics in a correct fashion.

I think that the problem lies with my setup, so if anyone has an idea on how to correct this setup in order to demonstrate the activation phase would be more than welcome.

0

There are 0 answers