Stopping the next execution of an Airflow DAG while the current execution is still running

Question

We are running Airflow on Google Cloud Composer (GCC) and we have a very long running task currently live that we want to finish. However, the schedule for the DAG is frequent enough that the next execution will start before this current one is finished.

If we pause the DAG to stop its next scheduled execution, will this also cause the current run to fail?

The end of these docs suggests that we are able to do this.

Unfortunately, we can't edit the code as the run is currently live, so the answer to this question is not possible.

Answer 1

Had a look at the Airflow docs themselves:

When a DAG is paused, any running tasks are allowed to complete and all downstream tasks are put in to a state of “Scheduled”. When the DAG is unpaused, any “scheduled” tasks will begin running according to the DAG logic. DAGs with no “scheduled” tasks will begin running according to their schedule.

So it is possible to pause a DAG to stop any new scheduled executions from running while waiting for the current execution to finish.

However, something to note, is that the task will finish not the entire DAG. So any follow up tasks to the task in question will be paused until the whole DAG is unpaused.

Answer 2

If you can just set the 'schedule_interval' parameter to 'None', you will prevent the next run. This does not change the running dag.