ODO cannot access S3 with S3ResponseError: 403 Forbidden

Question

I am using odo library to transfer a pandas dataframe to S3. However I am getting following error :

import pandas as pd
df = pd.DataFrame([[1, 2], [3, 4], [5, 6], [7, 8]], columns=["A", "B"])
odo(df,'s3://path_to_s3_folder')
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Anonymous access is forbidden for this operation</Message><RequestId>F5958774D56AD29E</RequestId><HostId>zOH8JOxpSgB5Scgc/YrtHO1+e9lXoKAF89IhRSeAiSoGHAJxyjXKBVFIYETeO4gSLZOUgXmwKLM=</HostId></Error>

Now I have the AWS credentials setup correctly as I can see in my ~/.aws/credentials file

cat credentials
[default]
aws_access_key_id = XXXXX
aws_secret_access_key = XXXXXXXXXX

The aws cli works correctly for me and I can run aws ls and cp commands correctly (I guess this means I do have the required permissions).

aws s3 ls s3://path_to_s3

Also boto3 is able to access s3 resources and does not give an error.

import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
    print(bucket.name)

What would be possible wrong/missing in the configuration ?

Answer 1

Please install boto.

After making a quick check on pandas odo source, the code only uses boto module, it does not support boto3.

Answer 2

As pointed by @mootmoot, odo uses boto and not boto3.

boto3 can get aws credentials from ~/.aws/credentials as stated here. However for boto you need to setup env variable as stated here

Just add,

$ export AWS_ACCESS_KEY_ID=XXXXX
$ export AWS_SECRET_ACCESS_KEY=XXXXXXXXXX