Keda with prometheus integration

Question

Below are the setup details: keda installed, prometheus deployed, through application using below scaled object template for hpa:

keda:
  triggers:
    - metadata:
        metricName: container_memory_usage
        query: avg(floor((container_memory_usage_bytes{pod=~"pythonb-.*",container=~"proxy"} / container_spec_memory_limit_bytes != +Inf)  * 100))
        serverAddress: <serveraddress>
        threshold: '70'
      type: prometheus

basically we want to scale the deployment based on the given prom query.(based on container memory utilisation of particular pods..if it exceeds 70% then hpa will scale the pods. ) when we try the above query on Prometheus it returns the results as 8., 10.. , 25.3. Basically single element response But though keda it gives the result as below:

kubectl get hpa -n integration keda-pythonb 
NAME                       REFERENCE                    TARGETS                               MINPODS   MAXPODS   REPLICAS   AGE
keda-pythonb   Deployment/pythonb   3500m/70 (avg), 34%/87% + 1 more...   2         10        2          14m

Instead of single value it gives 3500m as current value. does keda convert the data returned from prom query? Any pointers would be helpful. I hope the prom query is correct.

Answer 1

We just solved this one after a lot of off-and-on hunting. Turns out KEDA has an option called metricType that you can specify under triggers. TLDR you need to set that to "Value".

To understand why you need to dive into how HPA works in Kubernetes. When you define a kind: HorizontalPodAutoscaler you specify the metrics that are used for scaling. KEDA does this for you and creates an external metric like this:

  metrics:
  - external:
      metric:
        name: ...
        selector:
          matchLabels:
            scaledobject.keda.sh/name: ...
      target:
        type: AverageValue
        averageValue: ...
    type: External

There are Value and AverageValue metric types. AverageValue is the default, meant for metrics like http-requests-per-second, which would need to be divided by the number of replicas before compared to the target. Value, on the other hand, takes the direct value from your metric without averaging it.

Since your Prometheus query is returning an average across pods already, you need to use Value. The clue is in your kubectl get hpa output: 3500m/70 (avg).

See docs on HPA with external metrics.

In KEDA that is specified using the metricType option under the triggers field.

See KEDA: Scaling Deployments