Resource creation stuck as READY: False. How to inspect resource creation and trace errors in Crossplane?

Question

We have a Crossplane setup to create a ResourceGroup and StorageAccount in Azure (see fullblown example project on GitHub).

We use the official Azure Provider (meaning: the new Upbound split up provider families) provider-azure-storage and create the following crossplane manifests:

The Provider defintion:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-storage
spec:
  package: xpkg.upbound.io/upbound/provider-azure-storage:v0.39.0
  packagePullPolicy: Always
  revisionActivationPolicy: Automatic
  revisionHistoryLimit: 1

The ProviderConfig:

apiVersion: azure.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: azure-account-creds
      key: creds

The azure-account-creds are generated as described in the getting started guide.

Our CompositeResourceDefinition:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xstoragesazure.crossplane.jonashackt.io
spec:
  group: crossplane.jonashackt.io
  names:
    kind: XStorageAzure
    plural: xstoragesazure
  claimNames:
    kind: StorageAzure
    plural: storagesazure
  
  defaultCompositionRef:
    name: storageazure-composition

  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  location:
                    type: string
                  resourceGroupName:
                    type: string
                  storageAccountName:
                    type: string
                required:
                  - location
                  - resourceGroupName
                  - storageAccountName

Our Composition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: storageazure-composition
  labels:
    crossplane.io/xrd: xstoragesazure.crossplane.jonashackt.io
    provider: azure
spec:
  compositeTypeRef:
    apiVersion: crossplane.jonashackt.io/v1alpha1
    kind: XStorageAzure
  
  writeConnectionSecretsToNamespace: crossplane-system
  
  resources:
    - name: storageaccount
      base:
        apiVersion: storage.azure.upbound.io/v1beta1
        kind: Account
        metadata: {}
        spec:
          forProvider:
            accountKind: StorageV2
            accountTier: Standard
            accountReplicationType: LRS
      patches:
        - fromFieldPath: spec.parameters.storageAccountName
          toFieldPath: metadata.name
        - fromFieldPath: spec.parameters.resourceGroupName
          toFieldPath: spec.forProvider.resourceGroupName
        - fromFieldPath: spec.parameters.location
          toFieldPath: spec.forProvider.location
          
    - name: resourcegroup
      base:
        apiVersion: azure.upbound.io/v1beta1
        kind: ResourceGroup
        metadata: {}
      patches:
        - fromFieldPath: spec.parameters.resourceGroupName
          toFieldPath: metadata.name
        - fromFieldPath: spec.parameters.location
          toFieldPath: spec.forProvider.location

And finally our Claim:

apiVersion: crossplane.jonashackt.io/v1alpha1
kind: StorageAzure
metadata:
  namespace: default
  name: managed-storage-account
spec:
  compositionRef:
    name: storageazure-composition
  parameters:
    location: West Europe
    resourceGroupName: rg-crossplane
    storageAccountName: account4c8672f

Everything is applied via kubectl apply and doesn't throw any errors.

Although the StorageAccount has a READY: False when looking at kubectl get crossplane output.

How can we inspect the crossplane resources, so that we're able to trace down the error. The StorageAccount doesn't appear to be created in the Azure Portal.

Answer 1

There's a great documentation about how to troubleshoot your crossplane resources in the docs, which mainly focusses on running kubectl describe and kubectl get event on your crossplane resources.

But from crossplane 1.14 on there are new features included into the crossplane CLI. One of these features is the crossplane beta trace command, which allows to inspect any crossplane resources & see their respective status and aims to streamline the troubleshooting process:

trace to examine live resources and find the root cause of issues quickly

Now using the command in our example requires installation of the crossplane CLI first. According to the docs this could be achieved like this:

curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" | sh
sudo mv crossplane /usr/local/bin

Now having crossplane CLI in place we could use it to inspect our Claim's status via the following command:

crossplane beta trace StorageAzure managed-storage-account -o wide

After the trace you need to provide the name of your Claim (which is StorageAzure in our case) followed by the metadata.name (managed-storage-account here). The appended -o wide helps to see the full error messages if any.

Now this should provide you with valuable insights. We had this case where our Azure credentials were broken and now got a hint finally:

$ crossplane beta trace StorageAzure managed-storage-account -o wide
NAME                                             SYNCED   READY   STATUS                                                                                
StorageAzure/managed-storage-account (default)   True     False   Waiting: ...resource claim is waiting for composite resource to become Ready          
└─ XStorageAzure/managed-storage-account-6g2xn   True     False   Creating: Unready resources: resourcegroup, storageaccount                            
   ├─ Account/account4c8672d                     False    -       ReconcileError: ...r_uri":"https://login.microsoftonline.com/error?code=7000215"}:    
   └─ ResourceGroup/rg-crossplane                False    -       ReconcileError: ...r_uri":"https://login.microsoftonline.com/error?code=7000215"}: