Airflow Unittest.cfg permission problems?

Question

So, I'm setting up Airflow on EKS.
Everything seems to be in place, except that I cant use the KubernetesExecutor. Everytime I try to use it, I get this error:

with open(TEST_CONFIG_FILE, 'w') as f: PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/unittests.cfg'

I have tried going into the pods and chmod and chown the file. I tried the same in the docker image, and also I tried creating the unitttests.cfg file, and copying it over from my local to the image. They all return the same error.

Dockerimage:

FROM python:3.7.6-slim

ARG AIRFLOW_USER_HOME=/var/lib/airflow
ENV AIRFLOW_HOME=$AIRFLOW_USER_HOME

# Define en_US.
ENV LANGUAGE en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LC_CTYPE en_US.UTF-8
ENV LC_MESSAGES en_US.UTF-8

RUN mkdir $AIRFLOW_USER_HOME && \
  useradd -ms /bin/bash -d ${AIRFLOW_USER_HOME} airflow && \
  buildDeps='freetds-dev libkrb5-dev libsasl2-dev libssl-dev libffi-dev libpq-dev' \
  apt-get update && \
  apt-get install -yqq --no-install-recommends $buildDeps build-essential default-libmysqlclient-dev && \
  pip install psycopg2-binary \
  pip install --no-cache-dir 'apache-airflow[crypto,kubernetes,postgres,mysql]' && \
  apt-get purge --auto-remove -yqq $buildDeps && \
  apt-get autoremove -yqq --purge && \
  rm -rf /var/lib/apt/lists/*


COPY ./dags ${AIRFLOW_USER_HOME}/dags
COPY ./config/unittests.cfg ${AIRFLOW_USER_HOME}/unittests.cfg

RUN adduser airflow sudo

RUN chgrp -R airflow ${AIRFLOW_USER_HOME}
RUN chown -R airflow ${AIRFLOW_USER_HOME}

USER airflow

WORKDIR $AIRFLOW_USER_HOME

Dags:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime

default_args = {
    "start_date": datetime(2020, 9, 24),
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 0,
    'run_as_user': 1000
}

dag = DAG("test_bash_dag", default_args=default_args, schedule_interval=None, catchup=False)

t1 = BashOperator(task_id="foo", bash_command="echo foo", xcom_push=True, dag=dag)
t2 = BashOperator(task_id="bar", bash_command="echo bar", dag=dag)

t2.set_upstream(t1)

As you can see, the dag is literally just a test one, but I have been stuck here for a while.

Answer 1

Probably, it caused by absence of unittests.cfg.

Try to add: RUN airflow initdb (this command will create it)

Answer 2

I had a similar issue. My AIRFLOW_HOME env variable was pointing to the wrong directory, and fixed it by setting AIRFLOW_HOME env variable to the correct directory.

I think this is the case with OP's question, since the docker image sets ENV AIRFLOW_HOME="/var/lib/airflow", and yet the permission denied message implies that the AIRFLOW_HOME is set to "/usr/local/airflow/".

Somehow when executing airflow, it is loading a different AIRFLOW_HOME variable.

Answer 3

You can try extending airflow image https://airflow.apache.org/docs/apache-airflow/1.10.13/production-deployment.html#extending-the-image instead of installing airflow on python image.

Another tip - the official airflow helm chart https://airflow.apache.org/docs/helm-chart/stable/index.html is published, you can use it for Kubernetes executor.

Answer 4

I've been stuck at this problem too recently. Hopefully this helps. You need the same run user in your kubernetes settings (pod_template_file)

spec:
  containers:
    - args: []
      command: []
      env:
        - name: AIRFLOW__KUBERNETES__RUN_AS_USER
          value: 10000 # Or your user ID.

Make sure it's user ID not user name. For example, value 50000 works but value 'airflow' might not work.

In your case you ran this

RUN chown -R airflow ${AIRFLOW_USER_HOME}

So you need to figure out what is the user ID for this username 'airflow' and put that in pod template file like what I specified above in AIRFLOW__KUBERNETES__RUN_AS_USER environment variable.