AWS CloudFormation Template for Orchestration of mutliple AWS Glue Jobs (combination of sequentially and parallel execution)

Question

I'm looking for help on CloudFormation Template for Glue Jobs orchestration for below scenario:

Suppose I have 6 AWS Glue Jobs, 3 jobs (Job1, Job2, Job3) should be executed parallel and remaining 3 jobs should be executed sequentially (Job3 executed before Job4 then Job4 executed before Job5, then Job5 executed before Job6). If any job failure, send Workflow "Failure" notification along with the failed Glue Jobname.

---

Job1 
Job2 
job3 --->job4--->job5-->job6

---

Answer 1

You can define a StepFunctions using AWS::StepFunctions::StateMachine resource and define additionally AWS::Events::Rule for following events:

            EventPattern:
                source:
                    - "aws.states"
                detail-type:
                    - "Step Functions Execution Status Change"
                detail:
                    status:
                        - "FAILED"
                        - "TIMED_OUT"

Here is a sample single job execution stage in step functions:

                    Run Job 3:
                        Type: "Task"
                        Resource: "arn:aws:states:::glue:startJobRun.sync"
                        Parameters:
                            JobName: "GlueJob1Name"
                        Next: "Run Job 4"

You will have to enclose three parallel jobs within Parallel task type.