Run spark code serverless

Question

Hi All, what’s the quickest way to get started with running Spark code Serverless on AWS? Preferably through a notebook. Bonus points for the ability to run DynamicFrames, but not mandatory

Accepted Answer

The fastest way is to run workloads inside Glue ETL jobs, which is Spark.

You can connect SageMaker Notebooks with Glue Dev Endpoints. Then take that code out and put them into Glue ETL jobs.

Inside Glue, the Glue ETL library (DynamicFrames, etc.) will all be available.

Otherwise, the customer can also consider running EMR on EKS and connect with EMR Studio. However, that requires knowledge of managing EKS clusters and managing Fargate in EKS.

Answer

EMR serverless has just been announced at re:Invent, adding this as another option: https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/ (Note: its currently still in preview.)

Run spark code serverless

관련 콘텐츠