Concurrent jobs in emr serverless

0

Is there an option to run parallel jobs in emr serverless application natively

asked 9 months ago1.3K views
2 Answers
3
Accepted Answer

A bit of additional information.

As you can submit multiple jobs to an EMR-S application and each job will continue to run in parallel, however ensure those jobs hit the configured application limit or account level limit which has to be increased depends on your usecase and requirement. If you submit a single job with full capacity of application resource limit/account limit, then another job might not be executed in parallel.

Besides, to add more and more jobs with a fixed application maximum capacity limits, you need to handle them via AWS Step function and retry job submission in case the resources not available to serve the new jobs as of today.

To control the number of jobs that can run in parallel, we need the EMR serverless queue which can manage temporal overloads and force the jobs to wait for free resources when the application max capacity reached. This feature will be included in future soon.

profile pictureAWS
SUPPORT ENGINEER
answered 9 months ago
profile picture
EXPERT
reviewed 5 months ago
2

Yes, EMR Serverless supports running parallel jobs natively within a single application. It automatically manages resource allocation and scaling to accommodate concurrent execution of multiple jobs. You can get more information about it in the Announcing Amazon EMR Serverless

Announcing Amazon EMR Serverless

profile picture
EXPERT
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions