How to run existing EMR serverless job with boto3?

Question

From boto3 doc for the start_job_run, it seems like I have to create job run every time I want to trigger a job. Does it really have to work that way? Can't I take the ID of the existing job, which has already been defined with all the configuration it needs, and run it?

Reading the doc and searching on the internet

Answer 1

Yes that's the way it needs to be executed with boto3. You can call describe-job-run to gain the config information and then pass that to start-job-run.

This assumes that you have a short lived cluster. Executing jobs on a long lived cluster would be different.

Answer 2

EMR Serverless doesn't have "jobs" or templates (similar to EMR on EKS) where you can define all parameters and then reuse them for job runs, but only "job runs" themselves. So yes, you have to specify all parameters every time. You should be able to copy-paste job run config from another job run - use GetJobRun API for that.