Run spark code serverless
Hi All, what’s the quickest way to get started with running Spark code Serverless on AWS? Preferably through a notebook. Bonus points for the ability to run DynamicFrames, but not mandatory
- Más nuevo
- Más votos
- Más comentarios
The fastest way is to run workloads inside Glue ETL jobs, which is Spark.
You can connect SageMaker Notebooks with Glue Dev Endpoints. Then take that code out and put them into Glue ETL jobs.
Inside Glue, the Glue ETL library (DynamicFrames, etc.) will all be available.
Otherwise, the customer can also consider running EMR on EKS and connect with EMR Studio. However, that requires knowledge of managing EKS clusters and managing Fargate in EKS.
EMR serverless has just been announced at re:Invent, adding this as another option: https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/ (Note: its currently still in preview.)
Contenido relevante
OFICIAL DE AWSActualizada hace 3 años- OFICIAL DE AWSActualizada hace 2 años