aws elastic beanstalk cannot create scaling trigger fails with AWSEBCloudwatchAlarmHigh null values not allowed

Question

When I try to specify a scaling trigger it keeps erroring with

Service:AmazonCloudFormation, Message:[/Resources/AWSEBCloudwatchAlarmHigh/Type/Dimensions/0/Value/Fn::GetAtt/0] 'null' values are not allowed in templates

I have a saved template and I am trying to add

aws:autoscaling:trigger:
    BreachDuration: 5
    LowerBreachScaleIncrement: -1
    LowerThreshold: 0.75
    MeasureName: Latency
    Period: 1
    EvaluationPeriods: 1
    Statistic: Average
    Unit: Seconds
    UpperBreachScaleIncrement: 2
    UpperThreshold: 1

So I created it without this in which it created the auto alarm. I tried to update to this setting using the browser but it also failed with message

Service:AmazonCloudFormation, Message:[/Resources/AWSEBCloudwatchAlarmHigh/Type/Dimensions/0/Value/Fn::GetAtt/0] 'null' values are not allowed in templates

Here is my saved template

Platform:
  PlatformArn: arn:aws:elasticbeanstalk:eu-west-2::platform/Python 3.6 running on 64bit Amazon Linux/2.9.14
OptionSettings:
  aws:elasticbeanstalk:command:
    BatchSize: '30'
    BatchSizeType: Percentage
  AWSEBAutoScalingScaleUpPolicy.aws:autoscaling:trigger:
    UpperBreachScaleIncrement: '2'
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: domain.settings
    PYTHONPATH: $PYTHONPATH
    ALLOWED_CIDR_NETS: 10.0.0.0/16
    DATABASE_NAME: domainproductionplus
    DATABASE_HOST: domain-production-plus.coz8h02qupfe.eu-west-2.rds.amazonaws.com
    ENVIRONMENT: production
    DATABASE_PORT: '5432'
    EMAIL_BACKEND: django.core.mail.backends.console.EmailBackend
    DEBUG: '0'
    DATABASE_ENGINE: django.db.backends.postgresql_psycopg2
    REDIS_LOCATION: aws-co-qemfpydhs2ly.ubjsxm.0001.euw2.cache.amazonaws.com
    AWS_S3_REGION_NAME: eu-west-2
    ALLOWED_HOSTS: '*'
    VAPID_ADMIN_EMAIL: [email protected]
    DATABASE_USER: domainprodplus
    AWS_STORAGE_BUCKET_NAME: domain-production-plus
    REDIS_LOCATION_X: domain-production-plus-001.domain-production-plus.ubjsxm.euw2.cache.amazonaws.com
    DATABASE_PASSWORD: '{{resolve:ssm:domain-api-production-plus-DATABASE_PASSWORD:1}}'
    HASHID_SALT: '{{resolve:ssm:domain-api-production-plus-HASHID_SALT:1}}'
    VAPID_PRIVATE_KEY: '{{resolve:ssm:domain-api-production-plus-VAPID_PRIVATE_KEY:1}}'
    VAPID_PUBLIC_KEY: '{{resolve:ssm:domain-api-production-plus-VAPID_PUBLIC_KEY:1}}'
    SECRET_KEY: '{{resolve:ssm:domain-api-production-plus-SECRET_KEY:1}}'
    AWS_SECRET_ACCESS_KEY: '{{resolve:ssm:domain-api-production-plus-AWS_SECRET_ACCESS_KEY:1}}'
  aws:autoscaling:updatepolicy:rollingupdate:
    RollingUpdateType: Health
    RollingUpdateEnabled: true
  aws:elb:policies:
    domainionDrainingEnabled: true
  aws:ec2:instances:
    InstanceTypes: t2.micro
  AWSEBAutoScalingGroup.aws:autoscaling:asg:
    Cooldown: '120'
    MaxSize: '6'
  aws:elasticbeanstalk:container:python:
    WSGIPath: domain/wsgi.py
    StaticFiles: /static/=www/static/
  aws:ec2:vpc:
    VPCId: vpc-0fddefb70e6c8b32a
    Subnets: subnet-04497865d7eb17b70
    AssociatePublicIpAddress: false
  aws:elasticbeanstalk:environment:process:default:
    DeregistrationDelay: 20
    HealthCheckInterval: 15
    HealthCheckPath: /app-version-updates
    HealthCheckTimeout: 5
    HealthyThresholdCount: 3
    MatcherHTTPCode: 200
    Port: 80
    Protocol: HTTP
    StickinessEnabled: false
    StickinessLBCookieDuration: 86400
    StickinessType: lb_cookie
    UnhealthyThresholdCount: 5
  aws:elbv2:listener:80:
    ListenerEnabled: true
    Protocol: HTTP
    Rules: domainapiproductionplus
  aws:elbv2:listener:443:
    ListenerEnabled: true
    SSLCertificateArns: arn:aws:acm:eu-west-2:799479065523:certificate/5fb4f19c-f377-4ef6-8a7a-9657832c0d17
    Protocol: HTTPS
    Rules: domainapiproductionplus
    SSLPolicy: ELBSecurityPolicy-TLS-1-2-2017-01
  aws:elbv2:listenerrule:domainapiproductionplus:
    HostHeaders: api-production-plus.hcidomain.digital
    PathPatterns: /*
    Priority: 2
    process: default
  aws:elb:loadbalancer:
    CrossZone: true
    ManagedSecurityGroup: sg-0ac0850967d4d2929
  aws:elbv2:loadbalancer:
    ManagedSecurityGroup: sg-0ac0850967d4d2929
    SharedLoadBalancer: arn:aws:elasticloadbalancing:eu-west-2:799479065523:loadbalancer/app/domain-production-plus/206b8390c82843a3
  aws:elasticbeanstalk:environment:
    ServiceRole: arn:aws:iam::799479065523:role/aws-elasticbeanstalk-service-role
    LoadBalancerType: application
    LoadBalancerIsShared: true
  aws:autoscaling:launchconfiguration:
    IamInstanceProfile: aws-elasticbeanstalk-ec2-role
    EC2KeyName: domain
    SecurityGroups: sg-0ac0850967d4d2929,sg-095397beca170840e,sg-02f17712a24784d64
    MonitoringInterval: 1 minute
  aws:autoscaling:trigger:
    BreachDuration: 5
    LowerBreachScaleIncrement: -1
    LowerThreshold: 0.75
    MeasureName: Latency
    Period: 1
    EvaluationPeriods: 1
    Statistic: Average
    Unit: Seconds
    UpperBreachScaleIncrement: 2
    UpperThreshold: 1
  aws:elasticbeanstalk:healthreporting:system:
    SystemType: enhanced
EnvironmentTier:
  Type: Standard
  Name: WebServer
AWSConfigurationTemplateVersion: 1.1.0.0
Tags:
  project: domain
  product: domain

I am using a shared load balancer, could this be the issue? with classic load balancer it works fine - setting the autoscale metric to use Latency.

To create the environment from cli I run.

% eb create domain-api-production-plus --cfg domain-api-production-plus \
--cname domain-api-production-plus \
--elb-type application \
--shared-lb arn:aws:elasticloadbalancing:eu-west-2:799479065523:loadbalancer/app/domain-production-plus/206b8390c82843a3 \
--vpc \
--vpc.ec2subnets subnet-04497865d7eb17b70,subnet-032624d3e62d499f1 \
--vpc.elbsubnets subnet-0b3c3aa9b190a2546,subnet-05453d986413e8ae2 \
--vpc.id vpc-0fddefb70e6c8b32a \
--vpc.securitygroups sg-02f17712a24784d64,sg-095397beca170840e,sg-0ac0850967d4d2929 \
--tags project=domain,Name=domain-api-production-plus \
--service-role aws-elasticbeanstalk-service-role \
--region eu-west-2 \
--platform "arn:aws:elasticbeanstalk:eu-west-2::platform/Python 3.6 running on 64bit Amazon Linux/2.9.14" \
--keyname domain


Do you want to associate a public IP address? (Y/n): n
Do you want the load balancer to be public? (Select no for internal) (Y/n):
Creating application version archive "app-5aac-200929_084247".
Uploading Domain/app-5aac-200929_084247.zip to S3. This may take a while.
Upload Complete.
Environment details for: domain-api-production-plus
  Application name: Domain
  Region: eu-west-2
  Deployed Version: app-5aac-200929_084247
  Environment ID: e-tcwd2awzvs
  Platform: arn:aws:elasticbeanstalk:eu-west-2::platform/Python 3.6 running on 64bit Amazon Linux/2.9.14
  Tier: WebServer-Standard-1.0
  CNAME: domain-api-production-plus.eu-west-2.elasticbeanstalk.com
  Updated: 2020-09-29 07:42:50.765000+00:00
Printing Status:
2020-09-29 07:42:49    INFO    createEnvironment is starting.
2020-09-29 07:42:50    INFO    Using elasticbeanstalk-eu-west-2-799479065523 as Amazon S3 storage bucket for environment data.
2020-09-29 07:42:55    INFO    Created security group named: awseb-AWSEBManagedLBSecurityGroup-dw7edzemvt.
2020-09-29 07:43:13    INFO    Created target group named: arn:aws:elasticloadbalancing:eu-west-2:799479065523:targetgroup/awseb-domain--default-38qig/440561c9ab287e68
2020-09-29 07:43:13    INFO    Created security group named: sg-0e16398cbceab94d6
2020-09-29 07:43:14    INFO    Created Auto Scaling launch configuration named: awseb-e-tcwd2awzvs-stack-AWSEBAutoScalingLaunchConfiguration-1I6T492EE9NN1
2020-09-29 07:43:30    INFO    Created Load Balancer listener rule named: arn:aws:elasticloadbalancing:eu-west-2:799479065523:listener-rule/app/domain-production-plus/206b8390c82843a3/57e3715de5b24a02/0f045a9230df4191
2020-09-29 07:43:30    INFO    Created Load Balancer listener rule named: arn:aws:elasticloadbalancing:eu-west-2:799479065523:listener-rule/app/domain-production-plus/206b8390c82843a3/57e3715de5b24a02/f42374f1622dbd49
2020-09-29 07:43:30    INFO    Created Load Balancer listener rule named: arn:aws:elasticloadbalancing:eu-west-2:799479065523:listener-rule/app/domain-production-plus/206b8390c82843a3/457d5212b3cacc19/aa1f1cd2117f1290
2020-09-29 07:44:17    INFO    Created Auto Scaling group named: awseb-e-tcwd2awzvs-stack-AWSEBAutoScalingGroup-1I5GB63XJKP1Y
2020-09-29 07:44:17    INFO    Waiting for EC2 instances to launch. This may take a few minutes.
2020-09-29 07:44:17    INFO    Created Auto Scaling group policy named: arn:aws:autoscaling:eu-west-2:799479065523:scalingPolicy:664aebe1-ba1f-4d20-aed5-b44204b3a702:autoScalingGroupName/awseb-e-tcwd2awzvs-stack-AWSEBAutoScalingGroup-1I5GB63XJKP1Y:policyName/awseb-e-tcwd2awzvs-stack-AWSEBAutoScalingScaleDownPolicy-ZTC3D3FZQPZT
2020-09-29 07:44:17    INFO    Created Auto Scaling group policy named: arn:aws:autoscaling:eu-west-2:799479065523:scalingPolicy:c2431b1c-1efa-4927-a7ad-cdba75fa47ae:autoScalingGroupName/awseb-e-tcwd2awzvs-stack-AWSEBAutoScalingGroup-1I5GB63XJKP1Y:policyName/awseb-e-tcwd2awzvs-stack-AWSEBAutoScalingScaleUpPolicy-2MVWR52GOSTF
2020-09-29 07:44:32    INFO    Created CloudWatch alarm named: awseb-e-tcwd2awzvs-stack-AWSEBCloudwatchAlarmHigh-1OBABSCE98Y89
2020-09-29 07:44:32    INFO    Created CloudWatch alarm named: awseb-e-tcwd2awzvs-stack-AWSEBCloudwatchAlarmLow-UOJ9YBPCAIOH
2020-09-29 07:45:36    INFO    Successfully launched environment: domain-api-production-plus

UPDATE

So I cannot find the TargetResponseTime on the beanstalk environment.

Answer 1

I am using a shared load balancer, could this be the issue? with classic load balancer it works fine - setting the autoscale metric to use Latency..

Latency metric is only for CLB, not other load balancer types:

Latency: [HTTP listener] The total time elapsed, in seconds, from the time the load balancer sent the request to a registered instance until the instance started to send the response headers.

For the ALB, the closest metric would be:

TargetResponseTime: The time elapsed, in seconds, after the request leaves the load balancer until a response from the target is received. This is equivalent to the target_processing_time field in the access logs.

In your config file there is mixture of ALB and CLB settings. For example, aws:elb:loadbalancer is for CLB, while aws:elbv2:loadbalancer with SharedLoadBalancer is only for ALB.

Your aws:autoscaling:trigger is using Latency, which as explained above, is only for CLB. For ALB, it should be TargetResponseTime.

I can't verify if changing MeasureName in your ASG will solve all the issues you are having, but this is definitely a part which contributes to the problems.