Use custom docker image for EMR Serverless in a similar way to EMR on EKS

Question

Hello,

I am evaluating EMR Serverless and I have read in the documentation how you can package your Python dependencies into a virtual environment tarball and upload that to S3, then declare it in Spark config.

In contrast, for EMR on EKS you build a Docker image with your dependencies, upload it e.g. to ECR and specify it with `spark.kubernetes.container.image` option. This solution just seems more straightforward and natural than packaging virtualenvs in a .tar.gz... 
My question is, is something similar planned for EMR Serverless? Will we be able to just specify a custom Docker image in the future?

Accepted Answer

Hello there,

Happy New Year!

At the time of writing Custom Docker Image feature is not available on EMR Serverless. However, this has been a common requests originating from other customers and EMR Team had acknowledged the Feature Requests. Having said that, I do not have visibility into the release cycle of this feature and hence cannot advise on when this will be available. My best guess on this is that it would be released real soon, please follow the [release notes](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/release-versions.html) to keep you updated on when new releases happen.

Use custom docker image for EMR Serverless in a similar way to EMR on EKS

Relevant content