Want to push python file from Azure Repo to Databricks Workspace Using CICD

Question

I want to push the Python file from Azure Repo to Databricks Workspace using Azure DevOps (CICD), but when I am trying to push the same into the Databricks workspace it is converting the same python file into notebook, but I don't want the same.

I want that file to be the python file only.

Answer 1

To push a Python file from an Azure Repo to a Databricks Workspace without converting it to a notebook using Azure DevOps, you need to change your deployment method. The traditional deployment method depends on the Workspace REST API, which doesn't support arbitrary files.

Instead, you should have a repo in your destination workspace and update that repo via the Databricks CLI.

Yaml sample:

pool:
  vmImage: ubuntu-latest

steps:
- task: configuredatabricks@0
  inputs:
    url: '$(DATABRICKS_HOST)'
    token: '$(DATABRICKS_TOKEN)'

- script: |
    echo "Checking out the branch"
    databricks repos update --path /Repos/[email protected]/yourreponame --branch "$(Build.SourceBranchName)"
  env:
    DATABRICKS_HOST: $(DATABRICKS_HOST)
    DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
  displayName: 'Update databricks repository'

The pipeline :

The python file on databricks workspace repo:

You can check the similar ticket for more details.