A sane way to set up CloudWatch logs (awslogs-agent)

Question

tl;dr The configuration of cloudwatch agent is #$%^. Any straightforward way?

I wanted one place to store the logs, so I used Amazon CloudWatch Logs Agent. At first it seemed like I'd just add a Resource saying something like "create a log group, then a log stream and send this file, thank you" - all declarative and neat, but...

According to this doc I had to setup JSON configuration that created a BASH script that downloaded a Python script that set up the service that used a generated config in yet-another-language somewhere else.

I'd think logging is something frequently used, so there must be a declarative configuration way, not this 4-language crazy combo. Am I missing something, or is ops world so painful?

Thanks for ideas!

Answer 1

You've linked doco particular to CloudFormation so a bunch of the complexity is probably associated with that context.

Here's the stand-alone documentation for the Cloudwatch Logs Agent:

• Quick Start
• Agent Reference

If you're on Amazon Linux, you can install the 'awslogs' system package via yum. Once that's done, you can enable the logs plugin for the AWS CLI by making sure you have the following section in the CLI's config file:

[plugins]
cwlogs = cwlogs

E.g., the system package should create a file under /etc/awslogs/awscli.conf . You can use that file by setting the...

AWS_CONFIG_FILE=/etc/awslogs/awscli.conf

...environment variable.

Once that's all done, you can:

$ aws logs push help

and

$ cat /path/to/some/file | aws logs push [options]

The agent also comes with helpers to keep various log files in sync.

Answer 2

"Agent" is just an aws-cli plugin and a bunch of scripts. You can install the plugin with pip install awscli-cwlogs on most systems (assuming you already installed awscli itself). NOTE: I think Amazon Linux is not "most systems" and might require a different approach.

Then you'll need two configs: awscli config with the following content (also add credentials if needed and replace us-east-1 with your region):

[plugins]
cwlogs = cwlogs

[default]
region = us-east-1

and logging config with something like this (adjust to your needs according to the docs):

[general]
state_file = push-state
[logstream-cfn-init.log]
datetime_format = %Y-%m-%d %H:%M:%S,%f
file = /var/log/cfn-init.log
file_fingerprint_lines = 1-3
multi_line_start_pattern = {datetime_format}
log_group_name = ec2-logs
log_stream_name = {hostname}-{instance_id}/cfn-init.log
initial_position = start_of_file
encoding = utf_8
buffer_duration = 5000

after that, to start the daemon automatically you can create a systemd unit like this (change config paths to where you actually put them):

[Unit]
Description=CloudWatch logging daemon

[Service]
ExecStart=/usr/local/bin/aws logs push --config-file /etc/aws/cwlogs
Environment=AWS_CONFIG_FILE=/etc/aws/config
Restart=always
Type=simple

[Install]
WantedBy=multi-user.target

after that you can systemctl enable and systemctl start as usual. That's assuming your instance running a distribution that uses systemd (which is most of them nowadays but if not you should consult documentation to your distribution to learn how to run daemons).

Official setup script also adds a config for logrotate, I skipped that part because it wasn't required in my case but if your logs are rotated you might want to do something with it. Consult the setup script and logrotate documentation for details (essentially you just need to restart the daemon whenever files are rotated).