IBM Cloud Object Storage with VPC cluster

Question

I have been following this guide in order to add IBM Cloud object storage to my cluster.

While I have gotten it to work with a standard cluster, I have issues when trying to connect the COS to the VPC cluster. (Not the same object store, nor the cluster. Two completely different environments)

I have enabled VRF (and it has been approved) and I have created my own storageclass with the s3.direct endpoint.

However, the storage class seems to be unable to provision volume. The error I am getting is from TokenManagerRetrieveError: error retrieving the token. I was not able to find any documentation covering this case. I have checked the accounts IAM tokens and they are in place. Here are the events from the following command. See especially the second paragraph.

Here is the Storage Class that I have created:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cos-vpc
  namespace: default
parameters:    
  ibm.io/chunk-size-mb: "16"
  ibm.io/curl-debug: "false"
  ibm.io/debug-level: "warn"
  ibm.io/iam-endpoint: "https://iam.bluemix.net"
  ibm.io/kernel-cache: "true"
  ibm.io/multireq-max: "20"
  ibm.io/object-store-endpoint: "https://s3.direct.eu-de.cloud-object-storage.appdomain.cloud"
  ibm.io/object-store-storage-class: eu-de-standard
  ibm.io/parallel-count: "2"
  ibm.io/s3fs-fuse-retry-count: "5"
  ibm.io/stat-cache-size: "100000"
  ibm.io/tls-cipher-suite: AESGCM
provisioner: "ibm.io/ibmc-s3fs"
reclaimPolicy: Delete
volumeBindingMode: Immediate

And here is the pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hasher-java-pvc
  namespace: default
  annotations:
    ibm.io/auto-create-bucket: "false"
    ibm.io/auto-delete-bucket: "false"
    ibm.io/bucket: uia-bucket2
    ibm.io/secret-name: cos-write-access
spec:
  accessModes:
  - "ReadWriteOnce"
  resources:
    requests:
      storage: "13Gi"
  storageClassName: "cos-vpc"

Describing the pvc.

Name:          hasher-java-pvc
Namespace:     default
StorageClass:  cos-vpc
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   ibm.io/auto-create-bucket: false
               ibm.io/auto-delete-bucket: false
               ibm.io/bucket: uia-bucket2
               ibm.io/secret-name: cos-write-access
               volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    hasher-java-7c44896747-d5g8q
               hasher-java-7c44896747-gm2m9
               hasher-java-7c44896747-qhrxm
Events:
  Type     Reason                Age                    From                                                                                                  Message
  ----     ------                ----                   ----                                                                                                  -------
  Normal   Provisioning          4m52s (x3 over 5m53s)  ibm.io/ibmc-s3fs_ibmcloud-object-storage-plugin-d76dc998c-5wsx5_0f82cf0d-69e3-11ea-bdd1-2a0e250d0c0b  External provisioner is provisioning volume for claim "default/hasher-java-pvc"
  Warning  ProvisioningFailed    4m22s (x3 over 5m23s)  ibm.io/ibmc-s3fs_ibmcloud-object-storage-plugin-d76dc998c-5wsx5_0f82cf0d-69e3-11ea-bdd1-2a0e250d0c0b  failed to provision volume with StorageClass "cos-vpc": hasher-java-pvc:bppmei1f05qovjlv74vg:cannot access bucket uia-bucket2: TokenManagerRetrieveError: error retrieving the token
  Normal   ExternalProvisioning  12s (x24 over 5m53s)   persistentvolume-controller                                                                           waiting for a volume to be created, either by external provisioner "ibm.io/ibmc-s3fs" or manually created by system administrator

Any help is greatly appreciated.

Answer 1

This case is now listed under the troubleshooting section of the IBM Cloud docs here: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#cos_pvc_pending (scroll down to find match for this error)

It may need to create the secret to access the bucket using HMAC style of credentials instead of instanceid + IAM AKI key.

So try create credentials with HMAC keys as explained here: https://cloud.ibm.com/docs/containers?topic=containers-object_storage#create_cos_service

Then see the example for HMAC authentication where the docs explains how to create the secrets:

kubectl create secret generic cos-write-access --type=ibm/ibmc-s3fs --from-literal=access-key=<access_key_ID> --from-literal=secret-key=<secret_access_key>

(Reference doc: https://cloud.ibm.com/docs/containers?topic=containers-object_storage#create_cos_secret)