WALWriteLock wait elevated without any obvious cause

The unexpected spike in WALWriteLock wait times you observed during a period of typically low activity could be caused by several factors, even if the usual suspects like high IOPS usage or storage throughput limits weren't reached.

1. IO:WALWrite event: This event occurs when RDS for PostgreSQL is waiting for the write-ahead log (WAL) buffers to be written to a WAL file. An increase in this wait event could indicate issues with writing to the storage subsystem.

2. Storage latency: The spike in write latency you observed suggests there might be an underlying storage performance issue. Even if you're not hitting IOPS or throughput limits, there could be temporary storage system slowdowns or maintenance activities on the AWS side.

3. Background processes: While you mentioned that auto vacuum wasn't running and checkpoint activity appeared normal, other background processes might be contributing to the increased WAL write activity.

4. Network issues: Although less likely, network latency between the database and the storage system could potentially cause increased wait times for WAL writes.

5. Instance type limitations: The db.t3.xlarge instance type uses burstable performance. If you've exhausted your CPU credits, it could lead to performance degradation, including slower I/O operations.

6. Concurrent transactions: Even during low activity periods, if multiple transactions are committing simultaneously, they might contend for the WAL write lock.

7. AWS RDS issues: It's possible that there were underlying AWS RDS issues or maintenance activities that temporarily affected storage performance.

To further investigate:

1. Check AWS Health Dashboard for any reported issues in your region during that time.
2. Review your RDS logs for any error messages or warnings during the spike period.
3. Monitor your CPU credit balance to ensure you're not running out of burst capacity.
4. Consider enabling Performance Insights for more detailed analysis of database load.
5. If the issue persists or recurs, you might want to contact AWS support for a more in-depth investigation of potential underlying issues with the RDS instance or storage system.

Remember that occasional, short-lived performance spikes can occur in cloud environments due to various factors, some of which may be outside your direct control. If this becomes a recurring issue, you might need to consider upgrading your instance type or adjusting your storage configuration.
Sources
RDS for PostgreSQL wait events - Amazon Relational Database Service
Amazon Aurora PostgreSQL wait events - Amazon Aurora