Run spark code serverless
Hi All, what’s the quickest way to get started with running Spark code Serverless on AWS? Preferably through a notebook. Bonus points for the ability to run DynamicFrames, but not mandatory
- 最新
- 投票最多
- 评论最多
The fastest way is to run workloads inside Glue ETL jobs, which is Spark.
You can connect SageMaker Notebooks with Glue Dev Endpoints. Then take that code out and put them into Glue ETL jobs.
Inside Glue, the Glue ETL library (DynamicFrames, etc.) will all be available.
Otherwise, the customer can also consider running EMR on EKS and connect with EMR Studio. However, that requires knowledge of managing EKS clusters and managing Fargate in EKS.
EMR serverless has just been announced at re:Invent, adding this as another option: https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/ (Note: its currently still in preview.)
相关内容
AWS 官方已更新 1 年前
