how to add python library in airflow which is deployed by kubernetes

339 views Asked by At

I have deployed airflow via helmchart in minikube. I have one dag which runs dbtcloud job

After deploying airflow sucessfully, when I open my airflow UI, I receive this error on top of window:

File "", line 219, in _call_with_frames_removed File "/opt/airflow/dags/repo/example_dbt_cloud.py", line 22, in from airflow.providers.dbt.cloud.operators.dbt import ( ModuleNotFoundError: No module named 'airflow.providers.dbt'

I added below lines to resolve my issue in my deploy.sh file.

pip install apache-airflow-providers-dbt-cloud
pip install apache-airflow-providers-dbt-cloud[http]

But I assume, I added the library in the wrong place. I am not sure how I can have these library in my kuberneties. here is my deploy.sh file.

#!/bin/bash

source ./scripts/config.sh

# Create Resources from Templates
helm upgrade airflow-helper ./helperChart  \
    --install \
    --set env=${ENV} \
    --set cluster.namespace=${NAMESPACE} \
    -f ./helperChart/values.yaml

helm repo add apache-airflow https://airflow.apache.org
helm repo update

echo "Checking Previous Deployments"
if helm history --max 1 $RELEASE_NAME -n $NAMESPACE 2>/dev/null | grep -i FAILED | cut -f1 | grep -q 1; then
    echo "Deleting Airflow"
    helm uninstall  $RELEASE_NAME -n $NAMESPACE
fi

# Create Values File from Templated Values File
j2 ./yamls/values.yaml.j2 > ./yamls/values.yaml

unset KNOWN_HOSTS
pip install apache-airflow-providers-dbt-cloud
pip install apache-airflow-providers-dbt-cloud[http]
echo "Installing Airflow"
helm upgrade --install $RELEASE_NAME apache-airflow/airflow -n $NAMESPACE --create-namespace\
    -f ./yamls/values.yaml \
    --timeout=5m 
pip install apache-airflow-providers-dbt-cloud
pip install apache-airflow-providers-dbt-cloud[http]
rm ./yamls/values.yaml

Would you mind help me, if we are using a specific python library in a dag, how we should add it to the deployment process? I am new to kuberneties and helm.

additional info: I found this on airflow page but not sure how to integrate it with the current solution. :/

https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.htm

0

There are 0 answers