HorizontalPodAutoscaler scaleDown behavior

Question

We are using v1.18 and ArgoCD.

Desired Behavior: scale down by 1 pod at a time every 5 minutes when usage under 50%

The HPA scales up and down perfectly using default spec.

When we add the custom behavior to spec to achieve Desired Behavior, we do not see scaleDown happening at all.

I'm guessing that our configuration is in conflict with the algorithm and that this message may be a clue:

recent recommendations were higher than current one, applying the highest recent recommendation

EDIT: We got it working with the custom configured behavior with other settings such as 5 pods per 10 min. I've also noticed that in a policy value can be spread out across periodSeconds, meaning if my periodSeconds is 600 and value is 5, I can scale down 1 pod, then down 2 pods 120seconds later, and down 2 pods again 120seconds later, all within those 600 seconds whereas I had previously read that there will be exactly ONE scaling event of up to value number of pods per periodSeconds. Anyways, we are still trying to figure out how to scale down by 1 pod per 5 minutes. My theory is that if my current CPU avg is 49% with 10 pods, and I only allow scaling down by 1 pod at a time, we will end up with >50%(target) usage, so the "desired" pods stays the same.

HPA DEF

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: stackoverflow
  labels:
    app: question
  namespace: HPA
spec:
  scaleTargetRef:
    apiVersion: scaleTarget/v1alpha1
    kind: Rollout
    name: scaleTarget
  minReplicas: 2
  maxReplicas: 15
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300
      selectPolicy: Min

DESCRIBE HPA

Name:                                                  my-app
Namespace:                                             default
Labels:                                                app=my-app
                                                       app.kubernetes.io/instance=my-app
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration:
                                                         {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"app":"my-app","app.kubernete...
CreationTimestamp:                                     Wed, 20 Jan 2021 14:17:29 -0800
Reference:                                             Rollout/my-app
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  10% (10m) / 50%
Min replicas:                                          2
Max replicas:                                          15
Rollout pods:                                          15 current / 15 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    ScaleDownLimit       the desired replica count is decreasing faster than the maximum scale rate
Events:
  Type    Reason             Age                   From                       Message
  ----    ------             ----                  ----                       -------
 Normal  SuccessfulRescale  50m (x71 over 42h)  horizontal-pod-autoscaler  (combined from similar events): New size: 15; reason: cpu resource utilization (percentage of request) above target