By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Running Spark jobs on ECS

0

Is running Spark job on ECS (Fargate) recommended ? K8s has support for running spark drivers .. is there something similar for ECS ? I understand there are other services (Glue, EMR ) for Spark on AWS, but would like to understand the feasibility / complexity for running Spark on ECS Fargate.

Topics
ServerlessContainers
Tags
AWS FargateAmazon Elastic Container Service
Language
English
JackSparrow
asked 7 months ago1085 views
1 Answer
0

Hello.

Running Apache Spark on ECS (especially Fargate) is not a standard approach and isn't as streamlined as using EMR or Kubernetes. ECS is primarily designed for container orchestration of microservices, but with some effort, you can run Spark jobs on it.(check this article https://datachef.co/blog/run-spark-applications-on-aws-fargate/)

Regards, Andrii

profile picture
EXPERT
Andrii S
answered 7 months ago
  • JackSparrow
    7 months ago

    Thanks for the response. I understand it's not as streamlined as EMR or k8s. Having said that, I would specifically like to understand any known limitation / consideration to run Spark on ECS .

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content