I have a scenario where I want to use Helm to deploy per environment Airflow configurations that use the KubernetesExecutor.
The issue I am running into is I need to the Scheduler mount per environment ConfigMaps in the Worker Pods when the KubernetesExecutor spawns them.
I have those ConfigMaps mounted fine in the Web and Scheduler pods, but I don't see a means to tell the KubernetesExecutor to mount specific ConfigMaps.
I don't see anywhere in the documentation how to do this, but I can't be the only one who wants their Docker Images to be environment agnostic and abstract everything away to the Helm level.
Any suggestions would be much appreciated :)
Thanks!
I believe the easiest way to do this is by specifying the
env_from_configmap_ref
setting in yourairflow.cfg
as described here. If the configmaps are the same name in your different environments, this should be quite straight forward, e.g.If you have configmaps with different names in different environments (and can't make naming consistent), then you may need to specify the
airflow.cfg
in a configmap itself like in this example.