Run spark code serverless
Hi All, what’s the quickest way to get started with running Spark code Serverless on AWS? Preferably through a notebook. Bonus points for the ability to run DynamicFrames, but not mandatory
- 新しい順
- 投票が多い順
- コメントが多い順
The fastest way is to run workloads inside Glue ETL jobs, which is Spark.
You can connect SageMaker Notebooks with Glue Dev Endpoints. Then take that code out and put them into Glue ETL jobs.
Inside Glue, the Glue ETL library (DynamicFrames, etc.) will all be available.
Otherwise, the customer can also consider running EMR on EKS and connect with EMR Studio. However, that requires knowledge of managing EKS clusters and managing Fargate in EKS.
EMR serverless has just been announced at re:Invent, adding this as another option: https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/ (Note: its currently still in preview.)
関連するコンテンツ
AWS公式更新しました 1年前
