create object with python ibm-cos-sdk get doesn't work

Question

I need help... I can not access a bucket in my COS that I created with IBM Cloud dashboard from my notebook that I created in Watson Studio. Call this bucket3.

Initially I created a bucket2 from a single f2.zip (csv) file using IBM Watson 'Add Asset' and I am able to access the f2.zip. Call this bucket2.

The f2.zip was uploaded to bucket2 - later found out it was created on a previously COS storage; ie. asset created on IBM Cloud 'cloud-object-storage-nl'. Call this bucket1. I have a single Lite COS on IBM Cloud name = 'cloud-object-storage-xx'.

I can read the f2.zip and I am able to create a new f2.zip in bucket2 using the credential generated (cred_b2_editor) by the "Files UI" button.

IBM Cloud dashboard shows:

bucket1 us-geo Standard
bucket2 us-geo Standard
bucket3 us-east Standard

I am using the example from Creating a new text file in Using Python ibm-cos-sdk.

Fails with "ClientError: An error occurred (413) when calling the PutObject operation: Request Entity Too Large" on 2 cases:

1. when I use endpoint_url = 'endpoints' form the bucket credentials generated by IBM Cloud) - ibm_api_key_id does not matter.

Succeeds: when I use endpoint_url = 'endpoint_url' form the watson credentials generated by Watson Studio... writes to bucket2 regardless of ibm_api_key_id (bucket2 or bucket3)

CODE:

# Point to generated credentials
credDict = dict(b2 = cred_b2_editor,
                b3 = cred_b3_writer,
                watson = cred_watson
                )


bucketName = 'b3'
kwargs = dict(
    ibm_api_key_id=credDict[bucketName]['ibm_api_key_id'],
    ibm_service_instance_id=credDict[bucketName]['cred']['iam_serviceid_crn'], #COS_RESOURCE_CRN,
    ibm_auth_endpoint=COS_AUTH_ENDPOINT,
    config=Config(signature_version="oauth"),
    endpoint_url=credDict[bucketName]['ep_private']
    )
buckName = bucketDict[bucketName].split(':')[-1:][0]

print(buckName, kwargs['ibm_api_key_id'], kwargs['endpoint_url'])
cos = ibm_boto3.resource("s3", **kwargs)

#---> fix: bucketname needed to change with each bucket...
#---> fix: endpoint_url needs to point to private/public endpoint 
cos.Object(buckName, csvBN.replace('.csv','.zip')).put(
                Body=zbuf
            )

Credentials Code - all of the following were generated

'''
Cloud Resource Name or 'bucket ID string'

The last field is the `bucketName`
'''
bucketDict = dict(b2 = 'crn:v1:bluemix:public:cloud-object-storage:global:a/<IDNum>:<serviceID-seperated>:bucket:bucket2',
               b3 = 'crn:v1:bluemix:public:cloud-object-storage:global:a/<IDNum>:<serviceID-seperated>:bucket:bucket3'
              )

# Bucket2 Editor credentials - created by IBM Watson automatically
cred_b2_editor = {
  "apikey": "....",
  "cos_hmac_keys": {
    "access_key_id": "...",
    "secret_access_key": "..."
  },
  "endpoints": "https://cos-service.bluemix.net/endpoints",
  "iam_apikey_description": "Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloud-object-storage:global:a/<ServiceIDNum>:<COS_ID>::",
  "iam_apikey_name": "auto-generated-apikey-<apikey_2>",
  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/<ServiceIDNum>::serviceid:ServiceId-<serviceID_2>",
  "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/<ServiceIDNum>:<COS_ID>::"
}

## Bucket3 Create via IBM Cloud "New Credentials"
cred_b3_writer = {
  "apikey": "4hEJq-slh28Atvq3XnekZ4YOl0yWiv4LbFigoPS3oiuL",
  "endpoints": "https://cos-service.bluemix.net/endpoints",
  "iam_apikey_description": "Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloud-object-storage:global:a/<ServiceIDNum>-<COS_ID>::",
  "iam_apikey_name": "auto-generated-apikey-<apikey_3>",
  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/<ServiceIDNum>::serviceid:ServiceId-<ServiceID_3>",
  "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/<ServiceIDNum>:<COS_ID>::"
}

# Created inside juptyer notebook 10/01 button
cred_b2_cos = dict(ibm_api_key_id=cred_b2['apikey'],
                   ibm_auth_endpoint="https://iam.ng.bluemix.net/oidc/token",
                   config=Config(signature_version='oauth'),
                   ep_private='https://s3-api.us-geo.objectstorage.service.networklayer.com',
                   ep_public = 'https://s3-api.us-geo.objectstorage.softlayer.net',
                   cred = cred_b2_editor
                   )
cred_b3_cos = dict(ibm_api_key_id=cred_b3['apikey'],
                   ibm_auth_endpoint="https://iam.ng.bluemix.net/oidc/token",
                   config=Config(signature_version='oauth'),
                   ep_private = 'https://s3.us-east.objectstorage.service.networklayer.com',
                        ep_public = 'https://s3.us-east.objectstorage.softlayer.net',
                   cred = cred_b3_writer
                   )

Answer 1

The solution required:

1. import bucket.configuration.CRN which has bucketname
2. import a 'Writer' ServiceCredential, and
3. setting the bucketName and corresponding kwargs when calling cos.Object().

Key was setting the bucket endpoint_url to the private/public endpoint for the respective bucket.

The code examples were corrected to reflect changes.