How can I tell if my Amazon EC2 instance reaches its Amazon EBS quotas?

Short description

If your EC2 instance exceeds its Amazon EBS quotas, then you might encounter issues with EBS volume performance.

The following factors affect EBS volume performance:

• Amazon EBS performance quotas at the instance and volume level
• Input/output operations per second (IOPS) and throughput quotas
• Micro-bursting with brief moments of exceeded quotas that standard monitoring doesn't detect

Resolution

Make sure that you adhere to your performance quotas

Note: To avoid performance issues, make sure that your instance's device drivers are up to date.

An instance's Amazon EBS performance must adhere to two separate quotas that work together. Instance-level quotas define the maximum aggregate throughput and IOPS across all attached volumes. Volume-level quotas define the Provisioned IOPS and throughput for each volume. Your configuration adheres to whichever quota is lower. For more information, see Amazon EBS-optimized instance types.

To reach maximum Amazon EBS performance, the instance's attached volumes must provide combined performance equal to or greater than the maximum instance performance.

Review the bandwidth for your instance type to understand its IOPS and throughput quotas. IOPS typically affects workloads with many small files. Throughput typically affects workloads with large sequential read/write operations. To calculate throughput, multiply the IOPS by the block size.

Monitor Amazon EBS performance with CloudWatch

Use Amazon CloudWatch to monitor both instance-level and volume-level performance.

Monitor aggregate instance-level metrics

Note: The following instance-level metrics are available only for instances that are built on the AWS Nitro System. You can't use the following metrics for bare metal instances.

Check the following CloudWatch metrics in the EC2 namespace at the Per-Instance Metrics dimension:

• Check EBSReadBytes and EBSWriteBytes to monitor aggregate throughput usage.
• Check EBSReadOps and EBSWriteOps to monitor aggregate IOPS usage.
• Check EBSIOBalance% to monitor instance-level IOPS burst balance.
• Check EBSByteBalance% to monitor instance-level throughput burst balance.
• Check InstanceEBSIOPSExceededCheck to identify whether the instance exceeded its maximum aggregate IOPS quota.
• Check InstanceEBSThroughputExceededCheck to identify whether the instance exceeded its maximum aggregate throughput quota.

Monitor individual volume performance

Check the following CloudWatch metrics in the EBS namespace at the Per-Volume Metrics dimension:

• Check VolumeReadOps and VolumeWriteOps to track the number of read and write operations.
• Check VolumeReadBytes and VolumeWriteBytes to track the amount of data that the volume reads and writes.
• Check VolumeQueueLength to view the number of pending I/O requests.
  Note: This metric shows whether the volume is busy with pending I/O. If this value is high, then the IOPS is at or near its performance quota.
• Check VolumeThroughputExceededCheck to identify whether your volume exceeded the throughput quota.
• Check VolumeIOPSExceededCheck to identify whether your volume exceeded the IOPS quota.

Perform detailed diagnostics

For detailed metric calculations of the IOPS and throughput for the EC2 instance and each attached volume, use the AWSSupport-CalculateEBSPerformanceMetrics runbook.

Set up automated notifications for when volumes exceed performance quotas

Create a CloudWatch alarm to alert you when VolumeThroughputExceededCheck or VolumeIOPSExceededCheck is greater than 0. Also, create an alarm to notify you when EBSIOBalance% or EBSByteBalance% is less than 20%. For more information about the metrics, see Metrics for Nitro instances.

Identify and troubleshoot performance bottlenecks

Resolve instance-level bottlenecks

To identify whether your issue is at the instance level, check for the following issues:

• InstanceEBSIOPSExceededCheck or InstanceEBSThroughputExceededCheck show non-zero values.
• The combined values for EBSReadBytes and EBSWriteBytes are near or at the maximum instance bandwidth quotas.
• The EBSIOBalance% or EBSByteBalance% value is low.
  Note: This issue typically occurs if you depleted your burst credit balance.
• The VolumeQueueLength value is low on volumes that still experience performance issues.

To resolve instance-level issues, change to a larger instance type with higher Amazon EBS throughput and IOPS. Use EBS-optimized instance types to provide dedicated bandwidth to I/O for Amazon EBS.

Resolve volume-level bottlenecks

To identify whether your issue is at the volume level, check whether VolumeQueueLength is consistently above 1. Also, check whether VolumeThroughputExceededCheck or VolumeIOPSExceededCheck frequently show non-zero values.

To resolve volume-level issues, modify the EBS volume and increase the values for IOPS and Throughput.

Important: To avoid unnecessary delays because of EBS volume modification requirements, make all the required changes in a single modification request. This request can include changes to IOPS, throughput, size, and volume type.

If you have a workload with small files that uses high IOPS, then it's a best practice to increase your volume's IOPS. Or, compress the files before you copy them.

For Windows instances, run the following command to improve the performance of file copies:

robocopy source-folder destination-folder /MT:n

Note: Replace source-folder with the source folder name, destination-folder with the destination folder name, and n with the number of threads. For more information, see robocopy on the Microsoft website.

Detect micro-bursting

Micro-bursting occurs when your EBS volume exceeds its configured IOPS or throughput quotas for a short period.

The following issues might show that your instance experiences micro-bursting:

• Application with high latency
• CloudWatch metrics that show that performance is within your quotas
• Performance issues that occur intermittently

To troubleshoot this issue, see How do I identify whether my EBS volume experiences micro-bursting and make sure that it doesn't affect performance?

Related information

How do I troubleshoot EBS volume performance issues on my EC2 instance?

Amazon EBS volume types

Benchmark Amazon EBS volumes