Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

EMR serverless spark properties file

0

Hello,

How do we pass application.conf or properties file to spark jobs on emr serverless, in standard EMR we copy properties or conf files to master node /home/hadoop/?

Topics
AnalyticsServerless
Tags
ServerlessAmazon EMR ServerlessAmazon EMR
Language
English

asked 4 years ago1.2K views

1 Answer
0

You can place the file(s) in S3 and refer them using the standard --files parameter in spark parameters. The distinction in Serverless being if you intend to load this properties files and need to create an InputStream you will need to use SparkFiles.get(fileName) instead of just filename in the traditional EC2 based EMR cluster, read more about this here

AWS
SUPPORT ENGINEER

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content