RDS Read IOPS, Read Throughput, Queue Depth sudden increase
I got sudden spike in AWS RDS postgresql db.t3.small instances, it goes through 3000 Read IOPS, chugging all my IOPS Burst Balance until it empty and still stay on 300 IOPS, i've tried rebooting, increasing the gp2 size but no luck, upgrading the instance however partially mitigated the issue, as it doesn't chug all the way through burst balance.
Symptoms:
- Spike in read IOPS, read throughput, queue depth every 15 minute
- Started only 3 days ago
- When the spike occur, it also spike the cpu
I've tried:
- disabling autovacuum, but it only reduce write iops
- upgrading, to no avail, I am still on postgresql 10.17
System:
- AWS RDS with PostgreSQL 10.17
- some 2000-3000 databases ( we have database for each user )
- 170 GB gp2 usage
Monitoring: https://imgur.com/a/tPCDBUC
looks good as of 31 Jan
We're absolutely having the same problem over here: https://repost.aws/questions/QUIK9smsSVTriFJ3ynQ8w7iw#AN9SjEmXfuSmeQpjJhnhPF_w
We're reluctant to pay for AWS Support for something that seems like it's on AWS's end.
Additionally, we're now seeing the same thing on another database in the same AZ which is disconnected from the application. Started at just before midnight US/Eastern. Spiked to 3000 IOPS and almost drained all of our burst credits.
From what you described above, it may be an issue with underlying EBS volumes. Please have AWS support check in for you and raise support case if you have not already done so and we should be able to help by looking at your instance.
Should I get the developer support plan ?
GP2 with 170GB has a burst performance upto 3000 IOPS, and base performance of 170x3 = 510 IOPS. You can find detail here:
yes I know that, the problem isn't the base performance/burst capacity, its the spike that can go up to 3000 IOPS for a minute, every 15 minute, that start suddenly just 3 days ago
Relevant questions
RDS Read IOPS, Read Throughput, Queue Depth sudden increase
asked 4 months agoWhat is considered non-billable I/O
asked a month agoPostgres RDS Cant increase IOPS
asked 2 months agoBurstBalance drops to 0 during high DB write throughput
asked 3 years agoRDS high IOPS effects
asked 2 months agopoor MariaDB RDS gp2 storage volume write performance?
asked 5 months agoRDS instance running out of EBS byte balance
asked 3 months agoRDS MySQL Metrics
asked 3 years agoRDS Postgres + MySQL low read iops
asked 12 days agoPotentially degraded RDS volume - Latency & I/O spikes
asked 3 years ago
fikrimi, we seem to be seeing relief on our end. Wondered if you were seeing the same?