BurstBalance drops to 0 during high DB write throughput

Question

I noticed our application server becoming sluggish during certain periods and digging in to Cloudwatch has showed me that during these times our BurstBalance on our db.m5.large MySQL instance drops to zero.  During these times, CPU utilization is around 25%.  However Write IOPS are about 2500-3000 per second (while Read IOPS are only about 300 per second).  Also, the DB Write Throughput is 60 MBps.  
  
That last stat seems hard to account for, so I did some digging.  I enabled general logs on the DB instance and grabbed one of the larger log files to just take a peak at it.  In this log I see around 500k SELECT operations, 70k INSERT operations, 4K UPDATE operations and 14k DELETE operations.  Looking at the INSERT and UPDATES, I don't see how we could be getting 60Mbps DB Writes.  And it's odd that Cloudwatch is showing me 10x more write IOPS than Read IOPS when I have almost 10x more SELECT statements than INSERT and UPDATES.  Mysql isn't exactly my bag, but are there types of SELECT statements that would qualify as DB writes?  
  
Anywhere else I should be looking for the culprit?

Answer

Write throughput is not relevant. gp2 IOPS accounting is a function only of the IOPS used, not throughput.   
gp2 provides 5.4 million burst i/o credits at 3000 IOPS.   
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html  
  
When you use up this burst you will get throttled to baseline:   
https://aws.amazon.com/blogs/database/understanding-burst-vs-baseline-performance-with-amazon-rds-and-gp2/  
  
Your workload is clearly write heavy. The reason you see so few read IOPS is likely that many of these SELECTS are being served from the buffer cache and not disk. Your instance has 4 striped EBS volumes so while you have a very high number if burst iops, your workload is continually burning through this burst balance and then you get throttled to 1500 IOPS.  
  
To avoid getting throttled you could consider increasing the size of your allocated storage, since gp2 provides 3 IOPS per gb, you can do the math to allocate a volume to meet your minimum IOPS requirements. Or you could consider io1 and specify the precise IOPS you require  
  
-Phil

BurstBalance drops to 0 during high DB write throughput

Relevant content