Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

Run spark code serverless

1

Hi All, what’s the quickest way to get started with running Spark code Serverless on AWS? Preferably through a notebook. Bonus points for the ability to run DynamicFrames, but not mandatory

Argomenti
Serverless
Tag
Serverless
Lingua
English
AWS
MODERATORE
AWS-User-7984472
posta 3 anni fa484 visualizzazioni
2 Risposte
0
Risposta accettata

The fastest way is to run workloads inside Glue ETL jobs, which is Spark.

You can connect SageMaker Notebooks with Glue Dev Endpoints. Then take that code out and put them into Glue ETL jobs.

Inside Glue, the Glue ETL library (DynamicFrames, etc.) will all be available.

Otherwise, the customer can also consider running EMR on EKS and connect with EMR Studio. However, that requires knowledge of managing EKS clusters and managing Fargate in EKS.

AWS
Matthew Tan
con risposta 3 anni fa
0

EMR serverless has just been announced at re:Invent, adding this as another option: https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/ (Note: its currently still in preview.)

F2006
con risposta 2 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente