How do I reduce costs in Amazon EMR Serverless?

3 minute read

I want to reduce costs in Amazon EMR Serverless.

Resolution

Amazon EMR Serverless charges you for aggregate vCPU, memory, and storage resources that your workers use between the period they become ready to run and when they stop. EMR Serverless rounds up this time period to the nearest second with a one-minute minimum. To reduce costs in EMR Serverless, take the following actions.

Use Graviton on EMR Serverless

EMR Serverless provides x86_64 and arm64 architecture options for your applications. If you use AWS Graviton2 processors on EMR Serverless, then your costs might significantly decrease. For more information, see Using arm64 architecture (Graviton).

Avoid significant scaling in EMR Serverless

When you use Spark dynamic resource allocation, Apache Spark might excessively assign new workers. Spark prioritizes performance when it assigns new workers and this might increase costs. To prevent this, adjust spark.dynamicAllocation.maxExecutor to set your maximum threshold.

Optimize your resource usage in EMR Serverless

To optimize your resource usage in EMR Serverless, set dynamicAllocationOptimization to true in your Spark configuration classification. For more information, see Spark dynamic resource allocation optimization.

To maximize your cost efficiency, use either the job level setting spark.dynamicAllocation.maxExecutors or the application level maximum capacity setting. Use these settings to configure and upper scaling bound on workers.

Manage your disk space in workers for large scale jobs

Large scale jobs might use large data volumes with significant shuffles. As a result, your large scale jobs might require increased disk usage in EMR Serverless. Configurations such as spark.executor.cores combined with spark.emr-serverless.executor.disk and spark.dynamicAllocation.maxExecutors allow control over the associated worker size and total attached storage. To improve your performance for I/O intensive workloads, use Shuffle optimized disks.

With Amazon EMR versions 7.12.0 and later, you can also turn on serverless storage that eliminates local storage provisioning for Spark workloads. This reduces data processing costs and prevents job failures from disk capacity constraints. For more information, see Using serverless storage for Amazon EMR Serverless.

Optimize your infrastructure costs

When you use AWS Config, EMR Serverless creates an elastic network interface item record for every worker. To avoid costs related to this resource, turn off AWS::EC2::NetworkInterface in AWS Config. To avoid cross account Amazon Simple Storage Service (Amazon S3) data transfers, use an Amazon VPC endpoint for Amazon S3 instead of a NAT gateway.