Unable to launch a cluster in Azure Databricks

Question

Im triggering a Databricks job from Devops pipeline.It was working fine initially but after few runs the job is in pending state and failing with the below error.

run failed with error message
 Unexpected failure while waiting for the cluster (1031-182958-d705352n) to be ready: Cluster 1031-182958-d705352n is unusable since the cluster is unhealthy.

When I look at the event logs, I see the below error.

Message
Failed to add 1 container to the compute. Will attempt retry: true. Reason: Cloud provider launch failure

Help
A cloud provider error was encountered while launching worker nodes.

I checked the subscription level usage/quota, the Standard DSv2 Family vCPUs has 40% usage left.

Below is my configuration.

{
          "name": "'$jobName'",
          "new_cluster": {
            "spark_version": "14.0.x-scala2.12",
            "node_type_id": "Standard_DS3_v2",
            "num_workers": 0
            
          },
          "access_control_list": access_control_list,  
          "notebook_task": {
            "notebook_path": "'$notebookPath'",
            "base_parameters": {
              "env": "$(key)"
            }
          }

What could be the issue.

Thank you.

Answer 1

Is it working now? It could be cloud provider issue that affect certain regions. You may try check Service Health to see if any outage is happening next time:

Azure Service Health