How to stream custom logs to cloudwatch from AL2023 instances

Question

We use amazon-elastic-beanstalk to deploy part of our stack. We have just migrated our base platform from PHP 7.4 which uses Amazon Linux 2, to PHP 8.2 which uses Amazon Linux 2023.

Default system logs are being streamed properly (/var/log/nginx/access.log, /var/log/eb-hooks.log, etc.), but custom log streaming to cloudwatch is not working.

I followed this guide https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html in order to enable log streaming within amazon-linux-2

(The guide has not been updated to amazon-linux-2023)

In AL2 we declare the logs we want to be sent creating an ebextension that added the files within /etc/awslogs/config/logs.conf like described in this example from awsdocs

Also I had to enable Cloudwatch log streaming by creating another ebextension: .ebextension/95-logs-streamtocloudwatch.config

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

And finally I created a policy and add it to the role defined in the EC2 instance profile like documented in the official guide: Using Elastic Beanstalk with Amazon CloudWatch Logs

{
 "Version": "2012-10-17",
 "Statement": [
 {
   "Effect": "Allow",
   "Action": [
     "logs:CreateLogGroup",
     "logs:CreateLogStream",
     "logs:PutLogEvents",
     "logs:DescribeLogStreams"
   ],
   "Resource": [
   "*"
   ]
 }
 ]
}

from: official source

That configuration worked fine within amazon-linux-2 instances. But is not working within Amazon Linux 2023 instances

Answer 1

I have found the solution thanks to this existing answer

What I have done is to edit the .ebextension file in order to set the configuration for the cloudwatch agent under /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ folder.

This is my resulting .ebextension file

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

packages:
  yum:
    amazon-cloudwatch-agent: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`
  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/custom_logs.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/php-fpm/mycustomlog.log",
                  "log_group_name": "/aws/elasticbeanstalk/erp-prod-php8/var/log/php-fpm/mycustomlog.log",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }

commands:
  "01":
    command: systemctl enable amazon-cloudwatch-agent.service
  "02":
    command: systemctl restart amazon-cloudwatch-agent

Answer 2

The config mentioned above did not work for me. I created an cloudwatch config file using the commands mentioned in the other link Paul added as a reference and it worked! (attached logs). Also the cloudwatch agent is restarted as per the commands (see logs), but for some reason the logs don't stream until I manually ssh into the EC2 instance and run those 3 commands, again. Did anyone face this issue? If yes, appreciate if you can share how you solved it. Here's the config.

packages:
  yum:
    amazon-cloudwatch-agent: []
option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate  
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 90
files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`
  "/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/app/current/logs/spring.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/current/logs/spring.log"]]}`",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }
commands:
  "01":
    command: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json -s
  "02":
    command: sudo systemctl enable amazon-cloudwatch-agent.service
  "03":
    command: sudo systemctl restart amazon-cloudwatch-agent

and here's the result from the EB logs

2024-02-28 22:19:04,450 P38264 [INFO] ============================================================
2024-02-28 22:19:04,450 P38264 [INFO] yum list installed amazon-cloudwatch-agent
2024-02-28 22:19:04,763 P38264 [INFO] -----------------------Command Output-----------------------
2024-02-28 22:19:04,763 P38264 [INFO]   Installed Packages
2024-02-28 22:19:04,763 P38264 [INFO]   amazon-cloudwatch-agent.x86_64        1.300032.3-1.amzn2023         @amazonlinux
2024-02-28 22:19:04,763 P38264 [INFO] ------------------------------------------------------------
2024-02-28 22:19:04,763 P38264 [INFO] Completed successfully.
2024-02-28 22:19:04,766 P38264 [INFO] ============================================================
2024-02-28 22:19:04,766 P38264 [INFO] Command 01
2024-02-28 22:19:05,898 P38264 [INFO] -----------------------Command Output-----------------------
2024-02-28 22:19:05,898 P38264 [INFO]   ****** processing amazon-cloudwatch-agent ******
2024-02-28 22:19:05,898 P38264 [INFO]   I! Trying to detect region from ec2 D! [EC2] Found active network interface I! imds retry client will retry 1 timesSuccessfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_custom_logs.json.tmp
2024-02-28 22:19:05,898 P38264 [INFO]   Start configuration validation...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_beanstalk.json ...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_custom_logs.json.tmp ...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 I! Valid Json input schema.
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Configuration validation first phase succeeded
2024-02-28 22:19:05,898 P38264 [INFO]   /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2024-02-28 22:19:05,898 P38264 [INFO]   I! Detecting run_as_user...
2024-02-28 22:19:05,899 P38264 [INFO]   I! Trying to detect region from ec2
2024-02-28 22:19:05,899 P38264 [INFO]   D! [EC2] Found active network interface
2024-02-28 22:19:05,899 P38264 [INFO]   I! imds retry client will retry 1 times
2024-02-28 22:19:05,899 P38264 [INFO]   /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2024-02-28 22:19:05,899 P38264 [INFO]   Configuration validation second phase succeeded
2024-02-28 22:19:05,899 P38264 [INFO]   Configuration validation succeeded
2024-02-28 22:19:05,899 P38264 [INFO] ------------------------------------------------------------
2024-02-28 22:19:05,899 P38264 [INFO] Completed successfully.
2024-02-28 22:19:05,899 P38264 [INFO] ============================================================
2024-02-28 22:19:05,899 P38264 [INFO] Command 02
2024-02-28 22:19:06,292 P38264 [INFO] Completed successfully.
2024-02-28 22:19:06,293 P38264 [INFO] ============================================================
2024-02-28 22:19:06,293 P38264 [INFO] Command 03
2024-02-28 22:19:06,431 P38264 [INFO] Completed successfully.
2024-02-28 22:19:13,835 P38770 [INFO] ************************************************************

Edit:

Alright for those who are facing same problem, I figured it out. I had to use container_commands instead of commands. container_commands are executed after the app is deployed, at least that's how it is in AL2023. Updated config:

container_commands:
  01:
    command: "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json -s"
  02:
    command: "sudo systemctl enable amazon-cloudwatch-agent.service"
  03:
    command: "sudo systemctl restart amazon-cloudwatch-agent"