Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Environment variables for PySpark executor in AWS EMR Serverless and Env key limitations with EMR

0

Hello, I have gone documentation and practically observed the limitation for ENV Keys spark.emr-serverless.driverEnv and spark.emr-serverless.executorEnv with EMR Serverless which is limited to 50 max. I have more than 100 Env set on Code and need to pass to spark submit job. With EMR we were using --properties-file, How can we do it with EMR Serverless. Any help?

Topics
Analytics
Tags
Amazon EMR Serverless
Language
English
asked 2 years ago771 views
1 Answer
3
Accepted Answer

You can use custom AMI to add the env variables or dependencies required for your job in EMR serverless. More details here - https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html

AWS
SUPPORT ENGINEER
answered 2 years ago
EXPERT
reviewed 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content