- Newest
- Most votes
- Most comments
How many core nodes can a single primary node orchestrate?
The primary node in an EMR cluster is responsible for orchestrating the tasks and distributing them among core and task nodes. It doesn't directly limit the number of core nodes you can have. However, there might be practical limits based on network overhead, the capacity of the primary node to manage tasks, and other operational factors. AWS doesn't provide a specific maximum for the number of nodes, but clusters with thousands of nodes are not uncommon. Note that having a single primary node can become a bottleneck for extremely large clusters, but for most use cases, it's sufficient.
Availability commitment from AWS with a single primary node:
EMR does not provide a Service Level Agreement (SLA) for individual clusters. The primary node is a single point of failure in an EMR cluster. If it fails, the entire cluster becomes non-functional. Using 3 primary nodes increases the fault tolerance of the cluster as it can handle the failure of a primary node without losing the cluster. If you decide to go with a single primary node in your environment, you are sacrificing some fault tolerance.
Behavior when the primary node goes down:
If the primary node fails, the EMR cluster will become non-functional. EMR will not automatically recover a failed primary node. While you have termination protection enabled, it only prevents accidental termination through the AWS Management Console, the AWS SDKs, or the CLI. It doesn't provide protection against node failures. If a primary node goes down, you would typically need to terminate the cluster and start a new one. Your data stored in HDFS will be lost, but if you're using EMRFS with data in S3, that data remains intact.
To summarize, while you can use a single primary node for your EMR cluster, be aware of the potential risks associated with it. If fault tolerance is crucial for your lower environments, consider using multiple primary nodes.
Relevant content
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 5 months ago