Kubernetes autoscaling policies not working

Question

I've added some scaling policies to my HorizontalPodAutoscaler but they are not being applied. The scaleUp and scaleDown behaviours are being ignored. I need a way to stop pods scaling up and down every few minutes in response to small CPU spikes. Ideally the HPA would scale up quickly in response to more traffic but scale down slowly after about 30 minutes of reduced traffic.

I'm running this on an AWS EKS cluster and I have setup the policies according to https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior.

Could this be a limitation of EKS or my K8s version which is 1.14. I have run kubectl api-versions and my cluster does support autoscaling/v2beta2.

My Helm spec is:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ template "app.fullname" . }}
  labels:
    app: {{ template "app.name" . }}
    chart: {{ template "app.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: "{{ template "app.fullname" . }}-server"
  minReplicas: {{ .Values.hpa.minReplicas }}
  maxReplicas: {{ .Values.hpa.maxReplicas }}
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: AverageValue
        averageValue: 200m
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300
    scaleDown:
      stabilizationWindowSeconds: 1200
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300

Answer 1

As already discussed in the comments, even with autoscaling/v2beta2 enabled this function will not work on version 1.14.

Starting from v1.18 the v2beta2 API allows scaling behavior to be configured through the HPA behavior field.

The easiest way out of it would be to upgrade to 1.18.

Answer 2

i wanted to implement HPA based on targetAverageValue what worked for me

while GKE version is around 1.12 to 1.14 you wont be able to apply manifest of autoscaling/v2beta2 however you can apply same thing something like

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: core-deployment
  namespace: default
spec:
  maxReplicas: 9
  minReplicas: 5
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: core-deployment
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageValue: 500m

Answer 3

It works for me,

Client Version: v1.20.2 Server Version: v1.18.9-eks-d1db3c

kubectl api-versions and my cluster also supports autoscaling/v2beta2

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ template "ks.fullname" . }}-keycloak
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ template "ks.fullname" . }}-keycloak
  minReplicas: {{ .Values.keycloak.hpa.minpods }}
  maxReplicas: {{ .Values.keycloak.hpa.maxpods }}
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: {{ .Values.keycloak.hpa.memory.averageUtilization }}
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: {{ .Values.keycloak.hpa.cpu.averageUtilization }}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: {{ .Values.keycloak.hpa.stabilizationWindowSeconds }}
      policies:
        - type: Pods
          value: 1
          periodSeconds: {{ .Values.keycloak.hpa.periodSeconds }}
{{- end }}