I am trying to hit more than 10 Dataproc batches at a time in Serverless, which are automatically triggered from the Airflow, but only 5-6 batches are executing, and the remaining are getting failed due to insufficient quotas.
I have tried to change the number of workers from the configuration in Airflow, but couldn't solve it. Help me with necessary configuration edits in Dataproc and Airflow to run more than 10 Dataproc batches at a time.