Hello.
I have made an update in my Aurora mysql databases, after that, the read connection started to fail and to reboot.
This cycle of failures and restarts occurs when tried to run the next query:
LOAD DATA FROM s3 's3://xxxxxxx'
INTO TABLE temp_table FIELDS
TERMINATED BY '|'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (`id`)
Log from server
./grover/runtime/grover-volume.h:1028: Assertion failed: m_lsnManager != NULL: Stack trace:
<inline> (in Grover::Runtime::GroverVolume::getLsnManager() const at /local/p4clients/pkgbuild-fkx7d/workspace/src/GroverRuntime/./grover/runtime/grover-volume.h:1028)
/rdsdbbin/oscar/bin/mysqld() [0x19b1123] (in groverPartitionAllocateContiguousLsns at /local/p4clients/pkgbuild-fkx7d/workspace/src/GroverRuntime/grover/runtime/grover-csd.cpp:1458)
/rdsdbbin/oscar/bin/mysqld(_Z24log_allocate_grover_lsnsmPKmPP20_hgrover_write_statebb+0x153) [0x181a933]
/rdsdbbin/oscar/bin/mysqld(_Z35mtr_log_reserve_and_write_to_groverP5mtr_t+0xc1b) [0x182801b]
/rdsdbbin/oscar/bin/mysqld(_ZN5mtr_t6commitEv+0xcb) [0x16bddcb]
/rdsdbbin/oscar/bin/mysqld(_Z21dict_hdr_flush_row_idv+0xcb) [0x183b70b]
/rdsdbbin/oscar/bin/mysqld(_Z12row_ins_stepP9que_thr_t+0x5f7) [0x17284b7]
/rdsdbbin/oscar/bin/mysqld(_Z15que_run_threadsP9que_thr_t+0x4a7) [0x1702c07]
/rdsdbbin/oscar/bin/mysqld(_Z12que_eval_sqlP11pars_info_tPKcmP5trx_t+0x197) [0x1703657]
/rdsdbbin/oscar/bin/mysqld(_ZN8S3Stream20persistS3FileHistoryEmP5trx_t+0xe3) [0x1248ec3]
/rdsdbbin/oscar/bin/mysqld(_ZN8S3Stream22persistS3StreamHistoryEv+0x73) [0x1248ff3]
/rdsdbbin/oscar/bin/mysqld(_ZN8S3Stream4readEPhm+0x11f) [0x123b9bf]
/rdsdbbin/oscar/bin/mysqld(_my_b_s3_read+0xd3) [0x1156e33]
/rdsdbbin/oscar/bin/mysqld(_my_b_get+0x2f) [0x14b798f]
/rdsdbbin/oscar/bin/mysqld(_ZN9READ_INFO9next_lineEv+0x407) [0x126ef07]
/rdsdbbin/oscar/bin/mysqld(_Z10mysql_loadP3THDP12sql_exchangeP10TABLE_LISTR4ListI4ItemES8_S8_15enum_duplicates14LoadFileSource+0xa9f) [0x1272a3f]
/rdsdbbin/oscar/bin/mysqld(_Z30mysql_execute_command_internalP3THDb+0x1ba3) [0xfa5c43]
/rdsdbbin/oscar/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x12b) [0xfaa3ab]
/rdsdbbin/oscar/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3ab) [0xfab2eb]
/rdsdbbin/oscar/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1017) [0xfac497]
/rdsdbbin/oscar/bin/mysqld(_Z10do_commandP3THD+0x147) [0xfad107]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer7consumeEjjb+0x2a7) [0x10a15a7]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer5startEv+0x10f) [0x10a17af]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer11drain_queueEPv+0x4b) [0x10a19eb]
/rdsdbbin/oscar/bin/mysqld(pfs_spawn_thread+0xd7) [0x18dd437]
/lib64/libpthread.so.0(+0x72ab) [0x40001982a2ab]
/lib64/libc.so.6(+0xd5e5b) [0x4000212f2e5b]
03:14:59 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=9
max_threads=400
thread_count=9
connection_count=9
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 227952 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (401900cebae5): LOAD DATA FROM s3 's3://xxxxxx' INTO TABLE temp_table FIELDS TERMINATED BY '|' ENCLOSED BY '"' LINES TERMINATED BY ' ' IGNORE 1 LINES ( `id`)
Connection ID (thread ID): 18
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
aurora backtrace compare flag : 1
Writing a core file
2022-02-16T03:15:11.316356Z 0 [Warning] The syntax '--secure-auth' is deprecated and will be removed in a future release
2022-02-16T03:15:11.316576Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2022-02-16T03:15:11.316700Z 0 [Note] /rdsdbbin/oscar/bin/mysqld (mysqld 5.7.12) starting as process 19993 ...
2022-02-16T03:15:11.325755Z 0 [Warning] InnoDB: Setting innodb_checksums to OFF is DEPRECATED. This option may be removed in future releases. You should set innodb_checksum_algorithm=NONE instead.
2022-02-16T03:15:11.325795Z 0 [Note] InnoDB: Started in read only mode
2022-02-16T03:15:11.325803Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-02-16T03:15:11.325807Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-02-16T03:15:11.325810Z 0 [Note] InnoDB: Uses event mutexes
2022-02-16T03:15:11.325814Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-02-16T03:15:11.325817Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-02-16 03:15:11 0x400027042160[IB_LOG_LEVEL_INFO]:Initializing buffer pool, size = 89.7G (srv0start.cc:4406)
2022-02-16T03:15:11.325881Z 0 [Note] InnoDB: == Add SYNC_FAST DDL...
2022-02-16T03:15:11.326091Z 0 [Note] InnoDB: Number of pools: 1
2022-02-16T03:15:11.326217Z 0 [Note] InnoDB: Not using CPU crc32 instructions
2022-02-16T03:15:11.326224Z 0 [Note] InnoDB: Disabling background log and ibuf IO write threads.
2022-02-16T03:15:11.326944Z 0 [Note] InnoDB: Initializing buffer pool, total size = 89.7432G, instances = 16, chunk size = 5.60895G
2022-02-16 03:15:11 0x400027042160 Attempt to open data volume as READER
(buf0oscar.cc:799)
(buf0buf.cc:2411)
(buf0buf.cc:2411)
(buf0buf.cc:2411)
(buf0buf.cc:2411)
(buf0oscar.cc:799)
(buf0oscar.cc:799)
(buf0oscar.cc:799)
2022-02-16T03:15:13.952531Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-02-16T03:15:13.952807Z 0 [Note] InnoDB: page_cleaner coordinator priority: -20
2022-02-16T03:15:13.952815Z 0 [Note] InnoDB: page_cleaner worker priority: -20
2022-02-16T03:15:13.952825Z 0 [Note] InnoDB: page_cleaner worker priority: -20
2022-02-16T03:15:13.952825Z 0 [Note] InnoDB: page_cleaner worker priority: -20
2022-02-16T03:15:13.974660Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2022-02-16T03:15:14.009425Z 0 [Note] InnoDB: 5.7.12 started; log sequence number 0
2022-02-16T03:15:14.009612Z 0 [Note] Plugin 'FEDERATED' is disabled.
220216 3:15:14 server_audit: Audit STARTED.
Found DAS config file, trying to load DAS switcher from DAS config file.
2022-02-16 03:15:14 70369398759776:[DAS][INFO]: Calculated persistence threads 8
aurora_enable_das:0
220216 3:15:14 server_audit: server_audit_incl_users set to ''.
220216 3:15:14 server_audit: server_audit_excl_users set to ''.
2022-02-16T03:15:14.018244Z 0 [Warning] Read-only servers cannot create temp tables in InnoDB. Changing default engine for temp tables to MyISAM.
2022-02-16T03:15:14.018252Z 0 [Warning] Read-only servers cannot create intrinsic temp tables in InnoDB. Changing default engine for temp tables to MyISAM.
2022-02-16T03:15:14.019683Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
2022-02-16T03:15:14.020167Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2022-02-16T03:15:14.020229Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2022-02-16T03:15:14.020250Z 0 [Note] IPv6 is available.
2022-02-16T03:15:14.020258Z 0 [Note] - '::' resolves to '::';
2022-02-16T03:15:14.020373Z 0 [Note] Server socket created on IP: '::'.
2022-02-16T03:15:14.281071Z 0 [Note] /rdsdbbin/oscar/bin/mysqld: ready for connections.
Version: '5.7.12' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Server (GPL)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 11 100 11 0 0 11000 0 --:--:-- --:--:-- --:--:-- 11000
----------------------- END OF LOG ----------------------