All of the examples online seem to use boto. What code can I use to upload a file to s3 using boto3?
I'm trying to use the code from the official documentation, but I'm getting an error that version 4 authentication failed.
Put your keys in the keyfile according to the quickstart guide and then use the following code:
import boto3
s3 = boto3.resource(service_name='s3', region_name='REGION_NAME')
data = open('PATH_TO_FILE_ON_DISK', 'rb')
s3.Bucket('BUCKET_NAME').put_object(Key='FILENAME_ON_S3', Body=data)
The official documentation does not make it clear that the region name is required or else you may get an error, and it's not possible to store the configuration file for the region on heroku. It must be included in the resource call.
Since heroku doesn't have access to your AWS config file, you'll have to use environment variables. You can then use any of the various s3 file upload methods.
from os import getenv
import boto3
boto_kwargs = {
"aws_access_key_id": getenv("AWS_ACCESS_KEY_ID"),
"aws_secret_access_key": getenv("AWS_SECRET_ACCESS_KEY"),
"region_name": getenv("AWS_REGION"),
}
s3_client = boto3.Session(**boto_kwargs).client("s3")
s3_client.upload_fileobj(<f>, <bucket_name>, <object_name>)
Recently, Heroku released a good article on uploading files to S3 in Heroku. Direct to S3 File Uploads in Python.
According to instructions, you need to specify AWS Credentials and Bucket name in your Heroku App config vars.
In order for your application to access the AWS credentials for signing upload requests, they will need to be added as configuration variables in Heroku:
$ heroku config:set AWS_ACCESS_KEY_ID =xxx AWS_SECRET_ACCESS_KEY =yyy
Adding config vars and restarting app... done, v21
AWS_ACCESS_KEY_ID => xxx
AWS_SECRET_ACCESS_KEY => yyy
In addition to the AWS access credentials, set your target S3 bucket’s name:
heroku config:set S3_BUCKET = zzz
Adding config vars and restarting app... done, v21
S3_BUCKET => zzz
Note: if you are testing locally before deployment, remember to add the credentials to your local machine’s environment, too. It was described above how to make it with a config file.
Just for reference here's how you can fix the issue of v4 authentication being used for all new regions - set
signature_version
in the config file:Alternatively: