Data corruption with RDS MySQL 8.0.31

4

We recently upgraded to MySQL 8.0.31 on RDS. The upgrade went fine, but a couple days later the instance crashed because of data corruption and wouldn't restart. We ended up having to go boot in innodb_force_recovery, take a logical dump, and restore back to an older version.

Has anyone experienced a similar issue? This was not the same data corruption problem from 8.0.29 as we have not run any DDL since the upgrade, so it must be something new and unique to 8.0.31.

Here are some examples of the errors that started happening:

[MY-013183] [InnoDB] Assertion failure: btr0cur.cc:3660:rec thread 22349876958976

[MY-013183] [InnoDB] Assertion failure: row0ins.cc:267:!cursor->index->is_committed() thread 22766529312512

 [MY-012855] [InnoDB] Clustered record for sec rec not found index 

Once these errors start, the only fix is a point in time recovery back to before the upgrade or a logical dump of all data and import on a new instance.

BEWARE UPGRADING TO 8.0.31!

  • Hi. I hope this helps other people too like previous comments helped us. We have 3 envs under 8.0.28 for the last year. Always worked fine, until 2 weeks ago, when one day 3 people (including me) got a disconnected from server a couple of times one day, this using mysql workbench.

    But yesterday was chaos, 2 of the 3 envs (DEV and PROD) started disconnecting every 2 minutes or less, it was super odd.

    After many checks in our apis/elestic b/etc, we noticed the same message you posted here. AWS was saying one drop and create was the issue. PART1

  • PART2 - That statement has been there for months and never failed, also it is in stage and didn't fail.

    We had to force drop the sp that creates it because the super quick disconnect didn't allow us to do much, after this our connection didn't reset more. Never seen a db to disconnect just due to drop and create. It is a hardware issue in a sector of the disk that was corrupted, at least that's my impression.

    8.0.30 and 8.0.31 are not longer available to migrate into, meaning indeed they had issues, same as 8.0.28

preguntada hace un año2509 visualizaciones
3 Respuestas
2

We had 2 times the same problem here on mysql 8.0.31

DON'T UPGRADE TO 8.0.31 until AWS solve this.

LOG:

[ERROR] [MY-013183] [InnoDB] Assertion failure: btr0cur.cc:3660:rec thread 70398784480976
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2023-01-26T12:28:54Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x4006a5ceb000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 4006fe8a9e60 thread_stack 0x40000
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x30) [0x1f8f990]
/rdsdbbin/mysql/bin/mysqld(print_fatal_signal(int)+0x31c) [0xfb83fc]
/rdsdbbin/mysql/bin/mysqld(my_server_abort()+0x98) [0xfb85d8]
/rdsdbbin/mysql/bin/mysqld(my_abort()+0x14) [0x1f884f4]
/rdsdbbin/mysql/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x27c) [0x22ba11c]
/rdsdbbin/mysql/bin/mysqld(btr_cur_optimistic_update(unsigned long, btr_cur_t*, unsigned long**, mem_block_info_t**, upd_t const*, unsigned long, que_thr_t*, unsigned long, mtr_t*)+0x74c) [0x22fa4cc]
/rdsdbbin/mysql/bin/mysqld() [0x2486170]
/rdsdbbin/mysql/bin/mysqld() [0x24865fc]
/rdsdbbin/mysql/bin/mysqld(row_undo_mod(undo_node_t*, que_thr_t*)+0xcec) [0x2488c2c]
/rdsdbbin/mysql/bin/mysqld(row_undo_step(que_thr_t*)+0x58) [0x22449d8]
/rdsdbbin/mysql/bin/mysqld(que_run_threads(que_thr_t*)+0x688) [0x21f0188]
/rdsdbbin/mysql/bin/mysqld() [0x2299b2c]
/rdsdbbin/mysql/bin/mysqld() [0x229a8c0]
/rdsdbbin/mysql/bin/mysqld(trx_rollback_for_mysql(trx_t*)+0xd8) [0x229ccd8]
/rdsdbbin/mysql/bin/mysqld() [0x20f1f00]
/rdsdbbin/mysql/bin/mysqld(ha_rollback_low(THD*, bool)+0xf4) [0x10b3374]
/rdsdbbin/mysql/bin/mysqld(trx_coordinator::rollback_in_engines(THD*, bool)+0x30) [0xf68f10]
/rdsdbbin/mysql/bin/mysqld(MYSQL_BIN_LOG::rollback(THD*, bool)+0xec) [0x1be7f0c]
/rdsdbbin/mysql/bin/mysqld(ha_rollback_trans(THD*, bool)+0x90) [0x10b35b0]
/rdsdbbin/mysql/bin/mysqld(trans_rollback(THD*)+0x70) [0xf6c110]
/rdsdbbin/mysql/bin/mysqld(mysql_execute_command(THD*, bool)+0x2670) [0xe61b30]
/rdsdbbin/mysql/bin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x340) [0xe63380]
/rdsdbbin/mysql/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x15c0) [0xe64e60]
/rdsdbbin/mysql/bin/mysqld(do_command(THD*)+0x1e0) [0xe66160]
/rdsdbbin/mysql/bin/mysqld() [0xfaa2a8]
/rdsdbbin/mysql/bin/mysqld() [0x252fe90]
/lib64/libpthread.so.0(+0x722c) [0x40003b95f22c]
/lib64/libc.so.6(+0xd4a1c) [0x40003bb53a1c]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (4006dc34d030): rollback
Connection ID (thread ID): 51286
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
respondido hace un año
  • I haven't gotten any resolution from AWS on this issue yet. What I have gotten is a notice that AWS will be soon forcing older instances to be upgraded to 8.0.31 in a month or two, so this will become a much larger issue if they don't resolve it soon.

    I posted a but with MySQL here: https://bugs.mysql.com/bug.php?id=109677 but have yet to get a resolution from them either -- you may want to post your logs there too.

1

We just got this as well, thankfully on a staging database which was running 8.0.31 while our production is thankfully on an older minor version.

Yeah AWS guys gotta fix this for sure.

aditya
respondido hace un año
0

Hi

Currently there are no documented, well-known data corruption bugs for MySQL version 8.0.31.

If you have a specific error message - you may try looking for a specific bug via:

MySQL Bug Home - https://bugs.mysql.com/

AWS
respondido hace un año
  • Yes, that's why I'm posting here -- we would not have proceeded with the upgrade had there been known bugs.

    So there are clearly unknown data corruption bugs in MySQL 8.0.31 that people should be aware of. I've updated my question above with the specific error messages we started seeing.

  • Hello, Mysql Team are saying that this is a issue of AWS storage system can you solve the problem?

    Mysql team response: https://bugs.mysql.com/bug.php?id=109677

  • I've received confirmation from AWS that there is in fact a well-known data corruption bug that's been outstanding since November: https://github.com/mysql/mysql-server/commit/1070fe24851c41ac127bee2239b5c94296fcb464 -- this affected MySQL version 8.0.28 through 8.0.31 and is supposedly fixed in 8.0.32.

    Confusingly, the known bad versions 8.0.28, 8.0.30, and 8.0.31 are all still available on RDS.

  • this is very strange, I used version 8.0.28 for a long time months and months... I went to upgrade to 8.0.31 and in 1 week we had 2 incidents of corrupted data.

    It seems to me to be something else that occurs from this version 8.0.31

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas