RDS instance keeps rebooting

0

I have a MySQL RDS instance db.m4.xlarge, Multi-AZ, 10379 GiB Storage. 1100 GiB free space.
In this database I have 3 tables that are over 1 TB each and 30 or so tables of 1GB or much less.
It started throwing errors like this:

2019-02-02 18:31:53 2b740ec02700 InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 10621440:285212834, should be 552:17605632!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 17605632.
InnoDB: You may have to recover from a backup.

And rebooting the instance every few minutes.
I created a read replica, and promoted that replica, then moved all the traffic to that new instance, and removed the old one. That seemed to work for about a day, but then the errors returned.
I then created a snapshot, and restored that snapshot into a new instance, which worked for about a day, and then the errors returned.
I am accessing the database through JDBC. Using standard SQL queries, no triggers or stored procs.

I could use some suggestions.
Thanks!

Edited by: dustin8192 on Feb 2, 2019 11:07 AM

asked 5 years ago341 views
2 Answers
0

It seems likely that you have a data inconsistency in the innodb tables space. This gets propagated into a new read replica or a restore from backups. I'd recommend you use mysqldump to extract data from your instance and reload it back into a new instance to fix the metadata inconsistency.

-Phil

AWS
MODERATOR
philaws
answered 5 years ago
0

There was a corrupted index on a table. reading or writing a specific row in the table would cause the server to reboot. I created a new database from scratch from the code, and then transferred all records semi-manually using MANY mysqldump statements over several weeks - the db is 11TB.
No problems since.

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content