MWAA not Respecting any Concurrency Limits

36 views Asked by At

In MWAA, I have a great number of DAGs, requiring limiting the number of tasks and DAGs running at once. While MWAA respects the parallelism argument, it is not respecting core.dag_concurrency, core.max_tasks_per_dag, or the concurrency setting in my DAGs. All were set to 3. One task is run on every branch within my DAGs at once. Is this common? Is there a way to force MWAA to limit the number of tasks started to 3 instead of 1 task per branch?

For instance, if I have the following configuration and concurrency set to 3, 5 tasks start:

    dag_start >> task1 >>...>>dag_end
    dag_start >> task2 >>...>>dag_end
    dag_start >> task3 >>...>>dag_end
    dag_start >> task4 >>...>>dag_end
    dag_start >> task5 >>...>>dag_end

In the MWAA console, I set core.dag_concurrency and then core.max_tasks_per_dag (reverting back per the AWS docs). In the DAG, I set the concurrency argument.

0

There are 0 answers