botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request, could not connect to the endpoint URL

Question

I've read about this question, I have a similar issue, but by printing out the debug info, I got something slightly different, I'm not sure what I'm missing here:

When I run the following code, I always gets this error.

    s3 = boto3.resource('s3')
    bucket_name = "python-sdk-sample"
    print("Creating new bucket with name:", bucket_name)
    s3.create_bucket(Bucket=bucket_name)

I have saved my credential file in

C:\Users\myname\.aws\credentials, from where Boto should read my credentials.

Is my setting wrong?

Here is the output from boto3.set_stream_logger('botocore', level='DEBUG').

2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: env
2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: sso
2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2020-11-21 18:18:39,686 botocore.credentials [DEBUG] Looking for credentials via: config-file
2020-11-21 18:18:39,687 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2020-11-21 18:18:39,687 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2020-11-21 18:18:39,687 botocore.credentials [DEBUG] Looking for credentials via: container-role
2020-11-21 18:18:39,687 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2020-11-21 18:18:39,693 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://xxx.xxx.xxx.xxx/latest/api/token: Could not connect to the endpoint URL: "http://xxx.xxx.xxx.xxx/latest/api/token"

Answer 1

From my experience, this seems to occur when you've specified credentials incorrectly in your boto3 session, so it fails when communicating to AWS. That is, the profile name you've loaded in boto3 does not match a profile in either config or your credentials file. So for a config that looks like:

[profile dev]
us-east-2

The matching credentials profile should also contain [profile dev], and you load the session with:

import boto3
session = boto3.session.Session(profile_name='profile dev')

The error you indicated occurred for me when using:

import boto3
session = boto3.session.Session(profile_name='dev')

And for a completely incorrect name like 'foobar' the error message is a bit clearer:

ProfileNotFound: The config profile (foobar) could not be found