Mounting ConfigMap to Airflow Worker Pod with KubernetesExecutor

Question

I have a scenario where I want to use Helm to deploy per environment Airflow configurations that use the KubernetesExecutor.

The issue I am running into is I need to the Scheduler mount per environment ConfigMaps in the Worker Pods when the KubernetesExecutor spawns them.

I have those ConfigMaps mounted fine in the Web and Scheduler pods, but I don't see a means to tell the KubernetesExecutor to mount specific ConfigMaps.

I don't see anywhere in the documentation how to do this, but I can't be the only one who wants their Docker Images to be environment agnostic and abstract everything away to the Helm level.

Any suggestions would be much appreciated :)

Thanks!

Answer 1

I believe the easiest way to do this is by specifying the env_from_configmap_ref setting in your airflow.cfg as described here. If the configmaps are the same name in your different environments, this should be quite straight forward, e.g.

[kubernetes]
...
env_from_configmap_ref = <your-airflow-configmap>
...

If you have configmaps with different names in different environments (and can't make naming consistent), then you may need to specify the airflow.cfg in a configmap itself like in this example.