How do I configure automatic scaling in Amazon EMR?

Resolution

To configure automatic scaling in Amazon EMR, complete the following steps:

Amazon EMR versions 5.30.0, 6.1.0, and later

If you use Amazon EMR 5.30.0 and later (except Amazon EMR version 6.0.0), then use one of the following methods to configure automatic scaling:

• Turn on Amazon EMR managed scaling to automatically increase or decrease the number of instances or units in your cluster based on workload. For more information, see Configure managed scaling for Amazon EMR.
• Use automatic scaling with a custom policy for instance groups.

Amazon EMR versions 4.0.0 and later

Automatic scaling with a custom policy is available with the instance groups configuration and isn't available when you use instance fleets. For more information, see Using automatic scaling with a custom policy for instance groups.

To use Amazon CloudWatch metrics for automatic scaling in Amazon EMR, see Monitoring Amazon EMR metrics with CloudWatch. The following are recommended metrics to use for automatic scaling:

• YarnMemoryAvailablePercentage: This metric indicates the percentage of memory that's available for YARN.
• ContainerPendingRatio: This metric indicates the ratio of pending containers to allocated containers. Use this metric to scale a cluster based on container-allocation behavior for varied loads.

To confirm that the scaling policy is attached to the instance group, open the Amazon EMR console, and then choose Events from the left navigation pane. Then, check for automatic scaling policy events.

Related information

Use Amazon EMR cluster scaling to adjust for changing workloads