Why does Airflow best practice conflicts with PEP8 specifications?

Question

The Airflow documentation has certain best practices, one of which is to avoid top-level Python code. This is in direct contrast with the PEP8 specifications. Can someone please clarify?

Answer 1

The very introduction of PEP8 says that project-specific guidelines take precedence over PEP8:

Many projects have their own coding style guidelines. In the event of any conflicts, such project-specific guides take precedence for that project.

So, technically, Airflow does not conflict with PEP8 :) Also, just a few paragraphs later:

A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important.

However, know when to be inconsistent – sometimes style guide recommendations just aren’t applicable. When in doubt, use your best judgment. Look at other examples and decide what looks best. And don’t hesitate to ask!

The last paragraph is fairly critical: "sometimes style guide recommendations just aren’t applicable." In the case of Airflow, there is a very legitimate reason why top-level DAG code should be avoided, and it has nothing to do with style: all top-level Python code is executed when the DAG is parsed. Since DAGs are parsed very frequently (every 30 seconds by default), this can cause drastic performance degradation to the scheduler / DAG parsing process.

Import statements at the top of a file are top-level code, too, so if a module takes a long time to import (e.g. numpy), then it could adversely impact the scheduler, hence the advice to import these modules within the context of a task, but the vast majority of modules can be imported into Airflow DAGs at the top of the DAG file in compliance with PEP8 without issue. Honestly, I don't even think numpy is a problem, so I've opened a PR to update this section of the documentation with a more pragmatic example and description.