What is the cluster manager in SageMaker Spark Processing?

Question

SageMaker Processing can launch [multi-instance jobs][1]. What is the underlying [cluster manager][2]? Yarn? Mesos? Something custom?

[1]: https://aws.amazon.com/blogs/machine-learning/running-on-demand-serverless-apache-spark-data-processing-jobs-using-amazon-sagemaker-managed-spark-containers-and-the-amazon-sagemaker-sdk/
  [2]: https://spark.apache.org/docs/latest/cluster-overview.html

AWS-User-0466247 · Accepted Answer

The Spark container uses YARN - for ref the bootstrap script on github: https://github.com/aws/sagemaker-spark-container/blob/master/src/smspark/bootstrapper.py and the [Dockerfile](https://github.com/aws/sagemaker-spark-container/blob/master/spark/processing/2.4/py3/docker/Dockerfile.cpu) with hadoop-yarn dependencies

What is the cluster manager in SageMaker Spark Processing?

Add your answer

Relevant content