RDS instance keeps rebooting

Question

I have a MySQL RDS instance db.m4.xlarge, Multi-AZ, 10379 GiB Storage. 1100 GiB free space.  
In this database I have 3 tables that are over 1 TB each and 30 or so tables of 1GB or much less.  
It started throwing errors like this:  
  
2019-02-02 18:31:53 2b740ec02700 InnoDB: Error: space id and page n:o stored in the page  
InnoDB: read in are 10621440:285212834, should be 552:17605632!  
InnoDB: Database page corruption on disk or a failed  
InnoDB: file read of page 17605632.  
InnoDB: You may have to recover from a backup.  
  
And rebooting the instance every few minutes.   
I created a read replica, and promoted that replica, then moved all the traffic to that new instance, and removed the old one. That seemed to work for about a day, but then the errors returned.  
I then created a snapshot, and restored that snapshot into a new instance, which worked for about a day, and then the errors returned.  
I am accessing the database through JDBC. Using standard SQL queries, no triggers or stored procs.  
  
I could use some suggestions.  
Thanks!  
  
Edited by: dustin8192 on Feb 2, 2019 11:07 AM

Answer

There was a corrupted index on a table. reading or writing a specific row in the table would cause the server to reboot. I created a new database from scratch from the code, and then transferred all records semi-manually using MANY mysqldump statements over several weeks - the db is 11TB.   
No problems since.

Answer

It seems likely that you have a data inconsistency in the innodb tables space. This gets propagated into a new read replica or a restore from backups. I'd recommend you use mysqldump to extract data from your instance and reload it back into a new instance to fix the metadata inconsistency.  
  
-Phil

RDS instance keeps rebooting

相關內容