Data sharing between EMR Serverless Jobs

Question

Trying to share data between two spark jobs in an EMR serverless application using temp or global temp views without having to write to s3 and then do read. It doesn't seem to work.

What is the recommended approach?

Answer

In EMR Serverless you can **enable shared applications**. When you submit jobs, you can specify IAM role permissions that a EMR Serverless job run can assume when calling other services on your behalf. This includes access to Amazon S3 for any data sources, targets, as well as other AWS resources like Amazon Redshift clusters and DynamoDB tables. To learn more about how to create a role, see https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/getting-started.html#gs-runtime-role

Data sharing between EMR Serverless Jobs

Relevant content