Spark 2.4 on EMR Serverless

Question

Hi,
we have a task to move Spark jobs from on-prem to AWS, using lift-and-shift of the code as much as possible. Our Spark is based  on Scala 2.11 and Spark 2.4.0. I know EMR supports this version but we prefer serverless.

Will the job work with any version of EMR Serverless? I know the lower Spark version on EMR Serverless is 3.2.1, but wondering if backward compatibility works here?

acmanjon · Answer

Hi!

You will need to migrate your job to Spark 3.2.1 and Scala 2.12. Usually the changes needed are small or non-existent ( apart from library updates) from my experience.  You can see the differences here:

https://spark.apache.org/docs/latest/sql-migration-guide.html

Bests!

Spark 2.4 on EMR Serverless

Relevanter Inhalt