By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

EMR serverless spark properties file

0

Hello,

How do we pass application.conf or properties file to spark jobs on emr serverless, in standard EMR we copy properties or conf files to master node /home/hadoop/?

Topics
ServerlessAnalytics
Tags
ServerlessAmazon EMRAmazon EMR Serverless
Language
English
abhineet-s
asked 2 years ago841 views
1 Answer
0

You can place the file(s) in S3 and refer them using the standard --files parameter in spark parameters. The distinction in Serverless being if you intend to load this properties files and need to create an InputStream you will need to use SparkFiles.get(fileName) instead of just filename in the traditional EC2 based EMR cluster, read more about this here

AWS
SUPPORT ENGINEER
Krishnadas M
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content