We recently migrated our database to RDS MariaDB 10.6.12. Since then, our database randomly hangs. The symptom is:
- Multiple INSERTs or UPDATEs happening concurrently
- Normally, the queries each take less than a second, but when this happens, all queries are stuck taking many seconds, and never complete.
- Killing does not help; the queries will be stuck in the Killed state.
- All incoming INSERT/UPDATEs on any table, as well as SELECTs on the tables being written to, will also hang in the same way.
- After about 20 mins to 2 hours, the database crashes with this error in the log:
[ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
230523 22:10:10 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
RDS then restarts the database.
This happens randomly, almost once or twice a day, and has been happening for more than 2 weeks.
There are similar reports here for this version:
The comments in the last link suggest that this issue might be fixed in 10.6.13. Similarly, the comments in this ticket suggest that the hangs may be fixed in 10.6.13 as well.
MariaDB 10.6.13 was released on May 10, 2023, but as of today, almost a month later, it is still not yet available on RDS. Plus, 10.6.14 was just released yesterday as well.
We we would like to apply MariaDB 10.6.13 ASAP to fix the issue. Can anyone from AWS please confirm if MariaDB 10.6.13+ will be made available on RDS soon? The 10.6.12 version is currently unstable for us and affecting our products!
AWS OFFICIALUpdated 9 months ago
