RDS Latency High After Upgrade

Question

One of our SQL Server RDS instances (in us-east-2) underwent forced hardware maintenance this morning.

After maintenance completed, we began seeing substantially degraded latency from the database.

No other changes, potentially impacting or otherwise, were pushed during the window when the degradation began.

We reviewed documentation about the potential, temporary, performance penalties as a result of a database restoring from lazy-loaded S3 files (which presumably applies here), but we've seen no improvement in performance, despite a substantial period of time passing, and proactively taking measures, such as "SELECT *" on critical tables.

Is there anything critical step that we've missed?

Answer

There are a few things you can check based on the information provided:

* Verify the maintenance window for the RDS instance and check if any maintenance was scheduled during this time. Forced maintenance could cause performance impacts.
* Check the CloudWatch metrics for the database, particularly CPU utilization, to see if the instance is overprovisioned after the maintenance. Rebooting the instance may help if resources are not released.
* For SQL Server, review the SQL Server error log for any errors or warnings reported around the maintenance period.
* If using a single-AZ deployment, the storage volumes may need to be restored from backup after forced maintenance, causing temporary performance penalties until volumes are restored.
* For read replicas, check the replication lag metric to ensure the replica has caught up fully. Excessive lag can cause latency.
* Consider changing the instance type temporarily to a larger size to handle increased load until performance stabilizes again
* As a last resort, you can restore the database from the latest backup to fully refresh the data and storage.

RDS Latency High After Upgrade

Relevant content