Apache Airflow Multi Tenant

Question

I'm attempting to work out how Airflow might work in a multi-tenant environment. Specifically the requirements should look something like this:

1. Two teams, TeamA and TeamB, are using a single Airflow instance.
2. Team's A and B each have their own service user account: serviceUserA and ServiceUserB which they should run their jobs under.
3. For security reasons, Team A should not be able to create jobs that run under ServiceUserB and vice versa.

At this point it's not clear to me whether requirement 3. can be met with Airflow, short of giving each team a separate Airflow instance. Is there any way in which this can be achieved?

Answer 1

Until Airflow Version 1.9.0:

The reason the user is still able to access all the dags is that it is a superuser by default. Unless you use LDAP for authentication, all the users created are superusers and Airflow have no other roles.

However, if you use LDAP, you can have superuser and dataprofiler roles.

This should change in upcoming versions of Airflow.

As the documentation states here: You can use webserver:filter_by_owner only if you are using LDAP authentication. So to achieve multi-tenancy, you can use LDAP and give superuser permissions to only admins and have either user or data_profiler role for other users.

Answer 2

It depends what you mean with Airflow Instance.

You could spin up 2 groups of Airflow services (webserver, scheduler, worker) through systemctl, each of them relying on a different airflow.cfg.

In the airflow.cfg you would then separate the 2 groups.