I'm trying to deploy a Azure databricks instance integrated with Vnet on Azure Subscription. Vnet already exists

Question

I'm having an existing Vnet. I tried to follow approach from the below link. created two subnets(public and private) and NSGs and NSG Associations through terraform code and then use the custom_parameters block to provide the Network id and private_subnet_network_security_group_association_id. I'm deploying the code from Azure DevOps. It's throwing me errors:

creating/updating Workspace (Subscription: "xxxx-xxxx-xxx-xxx" Resource Group Name: "rg-xxxxx-test" Workspace Name: "xxxx-test-workspace"): polling after CreateOrUpdate: polling failed: the Azure API returned the following error: Status: "GatewayAuthenticationFailed" Code: "" Message: "Failed to prepare subnet 'xxxx-test-private'. Please try again later. Error details: 'Gateway authentication failed for 'Microsoft.Network'.

any clue on the above error?

text

I tried the below terraform code to create two subnets

resource "azurerm_subnet" "public" {
    name = "${var.dbname}-public-subnet"
    resource_group_name = data.azurerm_resource_group.qa.name
    virtual_network_name = data.azurerm_virtual_network.vnet.name
    address_prefixes = ["1.2.3.4/24"]

    delegation {
        name = "databricks_public"
        service_delegation {
            name = "Microsoft.Databricks/workspaces"
        }
    }
}

resource "azurerm_network_security_group" "nsg" {
    name = "${var.dbname}-qa-databricks-nsg"
    resource_group_name = data.azurerm_resource_group.qa.name
    location= data.azurerm_resource_group.qa.location
}

resource "azurerm_subnet_network_security_group_association" "nsga_public" {
    network_security_group_id = azurerm_network_security_group.nsg.id
    subnet_id = azurerm_subnet.public.id
}

and the custom_parameter block in databricks workspace creation.

 custom_parameters {
        public_subnet_name  = azurerm_subnet.public.name
        public_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.nsga_public.id
        private_subnet_name = azurerm_subnet.private.name
        private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.nsga_private.id
        virtual_network_id  = data.azurerm_virtual_network.vnet.id
    }

My subnets are getting created but while creation of workspace it is failing with errors.

Answer 1

I tired toI'm trying to deploy a Azure databricks instance integrated with Vnet on Azure Subscription using terraform and I was able to provision the requirement successfully

The error you're encountering, "GatewayAuthenticationFailed", typically indicates a problem with the network gateway, such as the Virtual Network Gateway or the Application Gateway within the Azure environment. This could be due to incorrect configuration, lack of permissions, or a failure in the service itself.

you're setting up subnets and associating them with a Network Security Group (NSG). The code structure appears to be correct for creating a subnet and an NSG, but there are a few points to consider:

1. Ensure that the Virtual Network (VNet) exists and is properly referenced by your Terraform data sources.
2. Verify that the address prefix for the subnet is within the address space of the VNet.
3. Check if there are any other services, like a VPN gateway or an ExpressRoute, associated with the subnet that may have authentication requirements.

My terraform configuration:

main.tf:

provider "azurerm" {
    features {}
}

data "azurerm_resource_group" "example" {
  name     = "demorg-vk"
}

resource "azurerm_virtual_network" "example" {
  name                = "vnet-demovk"
  address_space       = ["10.0.0.0/16"]
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
}

resource "azurerm_subnet" "public" {
  name                 = "public-subnet-vk"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/24"]

  service_endpoints = [
    "Microsoft.Storage",
    "Microsoft.Sql",
    "Microsoft.AzureActiveDirectory"
  ]

  delegation {
    name = "databricks_public"
    service_delegation {
      name    = "Microsoft.Databricks/workspaces"
      actions = ["Microsoft.Network/virtualNetworks/subnets/join/action"]
    }
  }
}

resource "azurerm_subnet" "private" {
  name                 = "private-subnet-vk"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]

  service_endpoints = [
    "Microsoft.Storage",
    "Microsoft.Sql"
  ]

  delegation {
    name = "databricks_private"
    service_delegation {
      name    = "Microsoft.Databricks/workspaces"
      actions = ["Microsoft.Network/virtualNetworks/subnets/join/action"]
    }
  }
}

resource "azurerm_network_security_group" "public_nsg" {
  name                = "nsg-public-demovk"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
}

resource "azurerm_network_security_group" "private_nsg" {
  name                = "nsg-demovk"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
}

resource "azurerm_subnet_network_security_group_association" "public_nsg_association" {
  subnet_id                 = azurerm_subnet.public.id
  network_security_group_id = azurerm_network_security_group.public_nsg.id
}

resource "azurerm_subnet_network_security_group_association" "private_nsg_association" {
  subnet_id                 = azurerm_subnet.private.id
  network_security_group_id = azurerm_network_security_group.private_nsg.id
}

resource "azurerm_databricks_workspace" "example" {
  name                = "databricks-example"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  sku                 = "standard"

  custom_parameters {
    no_public_ip                                  = false
    public_subnet_name                            = azurerm_subnet.public.name
    private_subnet_name                           = azurerm_subnet.private.name
    virtual_network_id                            = azurerm_virtual_network.example.id
    public_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.public_nsg_association.id
    private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.private_nsg_association.id
  }
}

Output: