Spark 2.4 on EMR Serverless

1

Hi, we have a task to move Spark jobs from on-prem to AWS, using lift-and-shift of the code as much as possible. Our Spark is based on Scala 2.11 and Spark 2.4.0. I know EMR supports this version but we prefer serverless.

Will the job work with any version of EMR Serverless? I know the lower Spark version on EMR Serverless is 3.2.1, but wondering if backward compatibility works here?

Michal
asked 10 months ago419 views
1 Answer
0

Hi!

You will need to migrate your job to Spark 3.2.1 and Scala 2.12. Usually the changes needed are small or non-existent ( apart from library updates) from my experience. You can see the differences here:

https://spark.apache.org/docs/latest/sql-migration-guide.html

Bests!

profile pictureAWS
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions