I have a branch operator and that completed successfully. It then entered the branch which starts with "create tables" task. But that task failed. The Update_watermarks task is setup to be run with trigger "none_failed_min_one_success". As the tasks have failed, I expect that the update_watermarks task would not have gotten invoked, and yet it did.
Am I misunderstanding the trigger? or is there a bug?
Airflow Documentation - Trigger Rules
none_failed_min_one_success: All upstream tasks have not failed or upstream_failed,
and at least one upstream task has succeeded.
Airflow Trigger Rules: All you need to know!
Oddly, I tried it with none_failed and it still allowed the task to complete:
And the description for none_failed is
none_failed: All upstream tasks have not failed or upstream_failed -
that is, all upstream tasks have succeeded or been skipped
Using all_success seems to provide partially the behavior I am looking for:
But the problem is the last task does not get executed, when the branch is skipped!
Ideally, it feels like none_failed_min_one_success should be what I want:
- If the branch operator or the tasks in the branch fail, then the last task should not run
- If the branch operator succeeds and the branch task succeed the last task should run
- If the branch operator succeeds and the branch tasks are skipped, it should still run the last task.