How do I troubleshoot an IOPS bottleneck in my Amazon RDS instance that causes latency in my Amazon EBS volumes?

Resolution

IOPS or throughput latency can occur in an Amazon RDS DB instance for the following reasons:

• An IOPS bottleneck at the instance level
• An IOPS bottleneck at the volume level
• A throughput bottleneck at the instance level
• A throughput bottleneck at the volume level
• Micro-bursting

Based on your latency issue, complete the following steps.

RDS instance with General Purpose SSD (gp2 and gp3)

To troubleshoot latency, first check the configuration information of the Amazon RDS instance, such as the DB instance class and storage size. You can use this information to track the IOPS and throughput quotas.

Note: To reduce issues with limits, it's a best practice to use gp3. For more information, see gp3 storage (recommended).

Use the CloudWatch graphs to review the following information and troubleshoot your issue:

• Check for spikes in the DiskQueueDepth, ReadLatency, and WriteLatency metrics. It's a best practice to use a DiskQueueDepth value of one per minute for every 1000 IOPS. ReadLatency and WriteLatency values must be less than 10 milliseconds. If you notice spikes in these metrics, then identify the time of the spike.
• After you identify the time of a spike, review the ReadIOPS and WriteIOPS metrics. Check if the IOPS limit was breached at the volume level during the timeframe of the spikes in the values of DiskQueueDepth, ReadLatency, and WriteLatency.
• If the IOPS limit breach occurs at the same time as the spikes, then check if there's a drop in the value of BurstBalance. This check is applicable only for GP2 volumes with a size of less than 1 TB. A drop in the value of BurstBalance confirms the occurrence of an IOPS bottleneck during the timeframe of the spike.
• Review the ReadThroughput and WriteThroughput metrics. Check if the throughput limit was breached at the volume level during the timeframe of the spikes in the values of ReadThroughput and WriteThroughput.
• If you use an EBS-optimized RDS instance class, then check for throttling of IOPS or throughput. For instance classes with burst capacity, view the EBSIOBalance% and EBSByteBalance% metrics in the CloudWatch graphs. Consistently low values of EBSIOBalance% or EBSByteBalance% indicate an IOPS or throughput bottleneck at the instance level.

Throttling of IOPS, throughput, or both indicates that the IOPS or throughput is inadequate for your workload at the storage level. To fix this issue, take the following actions:

• Locate the SQL queries that create more load on the database, and then optimize these queries. If the workload is as expected, or you can't tune the SQL queries, then increase the storage size to get a higher IOPS capacity.
  Note: After you increase the storage size of an RDS instance, you can't reduce the size to the previous value.
• Switch the volume from General Purpose (gp2) to Provisioned IOPS (io1) or io2. If the DB instance is Single-AZ and you use a custom parameter group, then switching between gp2 and io1 or io2 might cause a brief downtime. If your instance is Multi-AZ, then you don't experience any downtime.
• If you notice throttling of IOPS or throughput at the instance level, then scale up the instance class to get a higher IOPS or throughput capacity.

RDS instance with Provisioned IOPS (io1 and io2 Block Express)

To determine the IOPS or throughput limit, check the Amazon RDS instance configuration information, such as the DB instance class and defined Provisioned IOPS.

Note: To reduce issues with limits, it's a best practice to use io2 Block Express. For more information, see io2 Block Express storage (recommended).

Use the CloudWatch graphs to review the following information and troubleshoot your issue:

• Check for spikes in the DiskQueueDepth, ReadLatency, and WriteLatency metrics. It's a best practice to use a DiskQueueDepth value of one per minute for every 1000 IOPS. ReadLatency and WriteLatency values must be less than 10 milliseconds. If you notice spikes in these metrics, then identify the time of the spike.
• After you identify the time of a spike, review the ReadIOPS and WriteIOPS metrics. Check if the IOPS limit was breached at the volume level during the timeframe of the spikes in the values of DiskQueueDepth, ReadLatency, and WriteLatency.
• Review the ReadThroughput and WriteThroughput metrics. Check if the throughput limit was breached at the volume level during the timeframe of the spikes in the values of ReadThroughput and WriteThroughput.
• If you use an EBS-optimized RDS instance class, then check for throttling of IOPS or throughput. For instance classes with burst capacity, view the EBSIOBalance% and EBSByteBalance% metrics in the CloudWatch graphs. Consistently low values of EBSIOBalance% or EBSByteBalance% indicate an IOPS or throughput bottleneck at the instance level.

Throttling of IOPS, throughput, or both indicates that the IOPS or throughput is inadequate for your workload at the storage level. To fix this issue, take the following actions:

• Locate the SQL queries that create more load on the database, and then optimize these queries. If the workload is as expected, or you can't tune the SQL queries, then increase the IOPS. If you reach the maximum IOPS value of 64,000 for io1, then change to io2 for a higher limit. For more information on IOPS limits, see Amazon EBS volume types.
• If you notice throttling of IOPS or throughput at the instance level, then scale up the instance class to get a higher IOPS or throughput capacity.

Micro-bursting

Micro-bursting occurs when an EBS volume "bursts" high IOPS or throughput for significantly shorter periods than the collection period. CloudWatch metrics are collected at an interval of 60 seconds. Because the volume bursts high IOPS or throughput for a shorter time than the collection period, CloudWatch doesn't reflect the bursting. You can use Enhanced Monitoring to identify if microbursting causes the latency. Turn on Enhanced Monitoring with a granularity of 1 second. You can use the Read IO/s and Write IO/s metrics to determine the actual IOPS utilization. You can use the Read Kb/s and Write Kb/s to determine the actual throughput utilization per second. For more information, see Monitoring OS metrics with Enhanced Monitoring.

Related information

Monitoring Amazon RDS metrics with Amazon CloudWatch

Understanding burst vs. baseline performance with Amazon RDS and GP2

Hardware specifications for DB instance classes

How do I identify whether my EBS volume experiences micro-bursting and make sure that it doesn't affect performance?