Specifying IAM roles for permissions in AWS S3

Question

I'm trying to restrict all my AWS Cognito users to their own subdirectory in my S3 bucket.

I don't want them listing, reading, or writing other people's subdirectories/files in my larger bucket, and I only want them to read & write objects in their own directory.

I'm drawing inspiration from this AWS documentation snippet.

Here's my policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "subfolder/"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}",
                "arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
            ]
        }
    ]
}

And my code to retrieve the file of a certain user with user_id = [email protected], but actually allows me to retrieve a restricted file:

import boto

# These keys are *not* hardcoded, I'm just leaving out
# the auth flow to get them from Cognito/STS as described 
# here: https://mobile.awsblog.com/post/Tx2FL1QAPDE0UAH/Understanding-Amazon-Cognito-Authentication-Part-2-Developer-Authenticated-Ident
conn = boto.s3.connect_to_region('us-east-1',
    aws_access_key_id=ACCESS_KEY_FROM_COGNITO,
    aws_secret_access_key=SECRET_KEY_FROM_COGNITO,
    security_token=SECURITY_KEY_FROM_COGNITO)

# get the bucket
b = conn.get_bucket('my-bucket', validate=False)

# try to get an object we SHOULD be able to get
k = Key(b)
k.key = 'subfolder/us-east-1:xxxx-xxxx-xxxx-xxxxxxxxx/foobar'
print "Contents:", k.get_contents_as_string()  # success!

# try to get and object we SHOUDN'T be able to get
k2 = Key(b)
k2.key = 'subfolder/BLAH_BLAH/restricted'
print "Contents:", k2.get_contents_as_string()  # should fail, but doesn't

Unfortunately, I can access and read the contents of both files, yet I'm following the exact same pattern in the AWS blog documentation post. I'm also unsure why I need the validate=False in the boto connection, but it seems to work well enough.

What am I missing?

---

EDIT: In response to the answer below, I've tried updating my role to the following, but it does not make a difference:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "subfolder/${cognito-identity.amazonaws.com:sub}/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
            ]
        }
    ]
}

I've also confirmed that the access credentials I'm using are from Cognito by using the access/secret/security token triple retrieved from STS using Cognito token to create a boto IAMConnection object and querying for my role name corresponding to the auth'd cognito users for my identity pool. In doing so, I got the following exception when trying to read this role (which is exactly what should happen since I did not grant access):

BotoServerError: BotoServerError: 403 Forbidden
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <Error>
    <Type>Sender</Type>
    <Code>AccessDenied</Code>
    <Message>User: arn:aws:sts::MY_AWS_ACCT_ID:assumed-role/my_policy_role_name/session_name_here is not authorized to perform: iam:GetRole on resource: role awsNameFor_Role_Given_123012313</Message>
  </Error>
  <RequestId>xxx-xxxx-xxxx-xxxx-xxxxx</RequestId>
</ErrorResponse>

So still no clarity on why this isn't working.

Answer 1

5 things:

1. Make sure you are using credentials issued by the Amazon Cognito Identity service otherwise ${cognito-identity.amazonaws.com:sub} will be empty and grant you access to everything
2. Make sure the Amazon Cognito Identity credentials you are using were issued after you updated the policy, the policy is embedded in the session portion of the credentials so if you are using old credentials, they may not have the current policy attached.
3. You cannot use the username of the user, you must use the Amazon Cognito Identity id. So instead of [email protected] it will be the identity id: us-east-1:beef-beef-beef-xxxxxxxxx
4. Your pool has 2 roles associated with it, an unauthenticated role and an authenticated role. Make sure you are setting your policy on the correct role, in this case it looks like you are using developer authenticated identities and should be modifying the policy on the authenticated role.
5. Check your S3 bucket policy, if you allow anonymous access to your bucket, the Cognito role policy will not override it. Turn off anonymous access if that is the case. http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-2

Answer 2

It looks like you're code is using a hard-coded access key and secret key and is NOT using Cognito to retrieve credentials. Instead of embedding the same access key and secret key for all users, to leverage Cognito you'll need to follow the Authentication Flow and use GetId (boto reference, AWS reference) to get an identity ID and then GetCredentialsForIdentity (boto reference AWS reference) to get the Cognito-issued credentials for the current ID. Then use those credentials with the boto S3 connection.

Also be sure to cache the ID for each user and reuse it when making additional calls to Cognito.

Answer 3

The answer was rather silly. Apparently buckets themselves in S3 have their own policies (they are rather tucked away), but I had the directive:

Principal: "*"

which caused the bucket to be world readable.

The second revelation was that if you restrict a bucket with the s3:ListBucket with a Condition, that doesn't mean if you list the bucket you'll get only those results - you must call it by name. As an example in boto:

wrong = bucket.list()  # will simply 403
right = bucket.list(prefix="base/subdir/<cognito-id>/")  # will succeed

In other words S3 is designed such that you must know the prefix-key of the desired folder, which is good practice anyway.

I have to say, I was quite impressed with how helpful the folks at AWS were in diagnosing this issue here and on their forum. Anyway, a much better understanding of S3 now.