I am working on airflow migration to cloud composer task. The dags use Kubernetespodoperator to spin up a container and execute spark jobs.
In CC2, as the cluster is in autopilot mode, the pods get evicted due to ephemeral storage 1Gib default in autopilot mode.
My question is is there a way to find out, from the existing airflow, how much ephemeral storage a dag requires? I need this value to be set in limit_resources of the dag operator.
Similarly can i find the memory used by this particular dag from airflow 2?