AWS steps parallel state to orchestrate EMR jobs

Question

We are orchestrating data pipeline with AWS steps and we do need to run EMR jobs in parallel. I have tried using Map state and it works as expected. The only problem with Map is that in case one step fails , it cancels all the other steps as well. To overcome this issue , I am thinking if we can create an array of steps and pass it dynamically to Branches in parallel state but I have not been able to do it as it is not accepting strings. Is there a workaround for this or can we only hard code branches in Parallel state? Can States.Array() in someway be helpful in this situation?

Answer 1

Wrap the inner state machine in a one-branch parallel state and add error/retry policies to it. Basically, you want to catch all errors and ensure that the iteration always succeeds.

Answer 2

Just for someone who is trying to look for a solution to the stated problem. As suggested by Pooya, I did use catch block inside task within the Map rather than keeping it at map level.The state machine looks like this