Background: I have a separate Databricks Workspace for each environment, and I am buidling an Azure DevOps pipeline to deploy a Databricks Asset Bundles to these environments.
Question The asset bundle is configured in a databricks.yml file. How do I pass parameters to this file so I can change variables depending on the environment?
For example, I want to be able to vary my scheduling such that it runs more often in Prod than in the lower level environments. In the code below I want to pass {cronExpression} to the file from the pipeline.
Example from databricks.yml
:
bundle:
name: bundle_name
resources:
jobs:
job_name:
name: job_name
schedule:
quartz_cron_expression: {inputParameter}
tasks:
...
I have looked at Custom Variables, but it seems they need to be hard-coded in databricks.yml
. https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/settings#--custom-variables
There are a few methods of passing a value of a variable - it's all described in the documentation:
databricks.yaml
- then it will be the same for all environments (if they don't override it).BUNDLE_VAR_<variable_name>
--var="<var-name>=<var-value>"
But anyway you need to declare that variable before use - otherwise, the
bundle validate
command won't be able to resolve it & perform validation.If you want to do that from the Azure DevOps, then the last method could be the best, or maybe environment variables.