Spark 2.4 on EMR Serverless

Question

Hi,
we have a task to move Spark jobs from on-prem to AWS, using lift-and-shift of the code as much as possible. Our Spark is based  on Scala 2.11 and Spark 2.4.0. I know EMR supports this version but we prefer serverless.

Will the job work with any version of EMR Serverless? I know the lower Spark version on EMR Serverless is 3.2.1, but wondering if backward compatibility works here?

Answer

Hi!

You will need to migrate your job to Spark 3.2.1 and Scala 2.12. Usually the changes needed are small or non-existent ( apart from library updates) from my experience.  You can see the differences here:

https://spark.apache.org/docs/latest/sql-migration-guide.html

Bests!

Spark 2.4 on EMR Serverless

Contenido relevante