In MWAA, I have a great number of DAGs, requiring limiting the number of tasks and DAGs running at once. While MWAA respects the parallelism argument, it is not respecting core.dag_concurrency, core.max_tasks_per_dag, or the concurrency setting in my DAGs. All were set to 3. One task is run on every branch within my DAGs at once. Is this common? Is there a way to force MWAA to limit the number of tasks started to 3 instead of 1 task per branch?
For instance, if I have the following configuration and concurrency set to 3, 5 tasks start:
dag_start >> task1 >>...>>dag_end
dag_start >> task2 >>...>>dag_end
dag_start >> task3 >>...>>dag_end
dag_start >> task4 >>...>>dag_end
dag_start >> task5 >>...>>dag_end
In the MWAA console, I set core.dag_concurrency and then core.max_tasks_per_dag (reverting back per the AWS docs). In the DAG, I set the concurrency argument.