RDS Read IOPS, Read Throughput, Queue Depth sudden increase

Question

I got sudden spike in AWS RDS postgresql db.t3.small instances, it goes through 3000 Read IOPS, chugging all my IOPS Burst Balance until it empty and still stay on 300 IOPS, i've tried rebooting, increasing the gp2 size but no luck, upgrading the instance however partially mitigated the issue, as it doesn't chug all the way through burst balance.

Symptoms:
- Spike in read IOPS, read throughput, queue depth every 15 minute
- Started only 3 days ago
- When the spike occur, it also spike the cpu

I've tried:
- disabling autovacuum, but it only reduce write iops
- upgrading, to no avail, I am still on postgresql 10.17

System:
- AWS RDS with PostgreSQL 10.17
- some 2000-3000 databases ( we have database for each user )
- 170 GB gp2 usage

Monitoring:
https://imgur.com/a/tPCDBUC

Answer

Additionally, we're now seeing the same thing on another database in the same AZ which is disconnected from the application. Started at just before midnight US/Eastern. Spiked to 3000 IOPS and almost drained all of our burst credits.

Answer

We're absolutely having the same problem over here: https://repost.aws/questions/QUIK9smsSVTriFJ3ynQ8w7iw#AN9SjEmXfuSmeQpjJhnhPF_w

We're reluctant to pay for AWS Support for something that seems like it's on AWS's end.

Answer

Same case here, https://thumbs2.imgbox.com/ea/69/BCVUsMyC_t.png.

Answer

From what you described above, it may be an issue with underlying EBS volumes. Please have AWS support check in for you and raise support case if you have not already done so and we should be able to help by looking at your instance.

Answer

GP2 with 170GB has a burst performance upto 3000 IOPS, and base performance of 170x3 = 510 IOPS.  You can find detail here:
* [https://aws.amazon.com/blogs/database/understanding-burst-vs-baseline-performance-with-amazon-rds-and-gp2/]()

* [https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html]()

RDS Read IOPS, Read Throughput, Queue Depth sudden increase

관련 콘텐츠