Alarm when object size in S3 bucket exceeds threshold

Question

I have AWS data pipelines setup that feed to my S3 bucket. Each time a new feed file is generated by the pipeline and stored in the bucket. We keep at most 30 days of data in the bucket. Is it possible to configure an alarm so that I am notified via email, etc when the generated object size crosses the threshold (say 1G)? How would I go about it?

Answer 1

If you want granular data some dev work is required below are some options/further reading.

1. s3 notifications - ie events sent by s3 in response to create/delete etc which can be used to fire a lambda to perform whatever logic. You can base logic on key, filesize, created date etc. You could then store that value as a cloudwatch metric, and then setup an alarm on your custom metric.

See https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

Or

2. s3 inventory (which is basically a csv formatted directory listing uploaded to different bucket on a schedule).

If you go for inventory option you set a schedule and then you can then create a notification on destination bucket of inventory file to fire a lambda as each csv is availavle. Also take a look at aws Athena, can be used to query the inventory files direct via api - no need to download/parse csv!

See https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html

If your interested in quick n easy / none programming route there's a total bucket size cloudwatch metric called BucketSizeBytes which you could easily add an alarm which triggers sns email if total size got above 30gb. Depending on your goals this might be useful and should take minutes to setup - but is pretty useless for timely monitoring purposes.

See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/s3-metricscollected.html

Answer 2

• Navigate to AWS Cloudwatch Console -> Alarms -> All Alarms.
• Click Create Alarm.
• Click Select metric.
• Select S3 from AWS Namespaces.
• Select Storage Metrics.
• Find the s3 bucket you want, tick the one with BucketSizeBytes metric name.

• Then click Select metric.
• There you can configure your alarm.

Hope this helps!