List S3 buckets by bucket policies

Question

I have 400+ buckets in my AWS account some of which can be accessed by users using user group dev-user-group & prod-user-group. Few S3 buckets's policies are something like this

"aws:arn": [
     "arn:aws:sts::123XXXXX43:assumed-role/dev-user-group/*"
     "arn:aws:sts::123XXXXX43:assumed-role/prod-user-group/*"
]

Now, we would like to change it to the following

"aws:arn": [
     "arn:aws:sts::123XXXXX43:assumed-role/dev-eid/*"
     "arn:aws:sts::123XXXXX43:assumed-role/dev-p-eid/*"
     "arn:aws:sts::123XXXXX43:assumed-role/prod-eid/*"
     "arn:aws:sts::123XXXXX43:assumed-role/prod-p-eid/*"
]

Few buckets have only any one of accesses & few don't have any access. We would like to automate the process for updating the bucket policies using a script such that the script need to check if dev-user-group & prod-user-group is defined in the bucket policies. If so, it should remove them & add new policies.

I hope I conveyed better. Kindly give me suggestions on this.

Answer 1

You will need to create quick script and use the aws cli for this.

The script will first list all the buckets you have in the account aws s3 ls then save that list and loop over the list of buckets using this command which will output the policy as a json file:

aws s3api get-bucket-policy --bucket mybucket --query Policy --output text > policy.json

You can then modify the policy.json file as needed. Finally you can apply this modified policy back to the S3 bucket by running:

aws s3api put-bucket-policy --bucket mybucket --policy file://policy.json

Source

Answer 2

As a general rule:

• Use an IAM policy to assign bucket permissions to IAM entities (eg IAM Users, IAM Roles, IAM Groups), or
• Use an Amazon S3 bucket policy to grant public access to content

It is normally much easier to grant IAM entities permission by putting the policy on that IAM entity, rather than adding permissions on each S3 bucket.

Therefore, instead of granting access via a bucket policy on 400+ buckets, you could create add an IAM policy to the four IAM Roles (dev-eid, dev-p-eid, prod-eid, prod-p-eid) that grants them permissions on the buckets. It would look something like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::bucket1",
                "arn:aws:s3:::bucket2",
                "arn:aws:s3:::bucket3",
                "arn:aws:s3:::bucket4",
                "arn:aws:s3:::bucket5",
                "arn:aws:s3:::bucket6",
                "arn:aws:s3:::bucket7",
                "arn:aws:s3:::bucket8"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket1/*",
                "arn:aws:s3:::bucket2/*",
                "arn:aws:s3:::bucket3/*",
                "arn:aws:s3:::bucket4/*",
                "arn:aws:s3:::bucket5/*",
                "arn:aws:s3:::bucket6/*",
                "arn:aws:s3:::bucket7/*",
                "arn:aws:s3:::bucket8/*"
            ]
        }
    ]
}

You will likely face limitations due to the allowable size of policies, so multiple policies might be required (or it might not work at all, given you have so many buckets). A simpler method would be to reference bucket by prefix, such as:

"arn:aws:s3:::dev-*"

In this way, you can grant permission on any bucket that begins with dev-*, so the policy would actually be quite short (presuming that your buckets can be easily grouped in this way).