I have a DAG that collects data from an API. The data is a list of items that have different attributes aka a list of python dicts. Now I want to loop thru the data and start an another DAG with the item as parameter, one instance per item. I don't want to use subtags to avoid any problems with the worker limit and deadlocks. Also it does not matter for the DAG that fetches the data form the API if the individual DAG runs per item fail or not. The DAGs per item can all start at the same time and run in parallel, if sufficient workers are available.
I am for the best way and a code example how the loop would be build.
I thought about a function called by a python operator that loops thru the dataset then calls a DAG run operator with every iteration with the item as parameter.
But I could not find any examples for this use case. I am also interested if this is the best way to do it or if there is a better way. Important to me is that this scales. If the list contained 10 or 1000 objects, I have to avoid to get into a deadlock when I run out of workers.