I have deployed airflow via helmchart in minikube. I have one dag which runs dbtcloud job
After deploying airflow sucessfully, when I open my airflow UI, I receive this error on top of window:
File "", line 219, in _call_with_frames_removed File "/opt/airflow/dags/repo/example_dbt_cloud.py", line 22, in from airflow.providers.dbt.cloud.operators.dbt import ( ModuleNotFoundError: No module named 'airflow.providers.dbt'
I added below lines to resolve my issue in my deploy.sh file.
pip install apache-airflow-providers-dbt-cloud
pip install apache-airflow-providers-dbt-cloud[http]
But I assume, I added the library in the wrong place. I am not sure how I can have these library in my kuberneties. here is my deploy.sh file.
#!/bin/bash
source ./scripts/config.sh
# Create Resources from Templates
helm upgrade airflow-helper ./helperChart \
--install \
--set env=${ENV} \
--set cluster.namespace=${NAMESPACE} \
-f ./helperChart/values.yaml
helm repo add apache-airflow https://airflow.apache.org
helm repo update
echo "Checking Previous Deployments"
if helm history --max 1 $RELEASE_NAME -n $NAMESPACE 2>/dev/null | grep -i FAILED | cut -f1 | grep -q 1; then
echo "Deleting Airflow"
helm uninstall $RELEASE_NAME -n $NAMESPACE
fi
# Create Values File from Templated Values File
j2 ./yamls/values.yaml.j2 > ./yamls/values.yaml
unset KNOWN_HOSTS
pip install apache-airflow-providers-dbt-cloud
pip install apache-airflow-providers-dbt-cloud[http]
echo "Installing Airflow"
helm upgrade --install $RELEASE_NAME apache-airflow/airflow -n $NAMESPACE --create-namespace\
-f ./yamls/values.yaml \
--timeout=5m
pip install apache-airflow-providers-dbt-cloud
pip install apache-airflow-providers-dbt-cloud[http]
rm ./yamls/values.yaml
Would you mind help me, if we are using a specific python library in a dag, how we should add it to the deployment process? I am new to kuberneties and helm.
additional info: I found this on airflow page but not sure how to integrate it with the current solution. :/