Upgrading from Aurora 1.22.2 to 2.10.2 [URGENT]

1

Hello.

I have made an update in my Aurora mysql databases, after that, the read connection started to fail and to reboot.

This cycle of failures and restarts occurs when tried to run the next query:

LOAD DATA FROM s3 's3://xxxxxxx' 
INTO TABLE temp_table FIELDS 
TERMINATED BY '|' 
ENCLOSED BY '"' 
LINES TERMINATED BY '\n' 
IGNORE 1 LINES (`id`)

Log from server

./grover/runtime/grover-volume.h:1028: Assertion failed: m_lsnManager != NULL: Stack trace:
<inline> (in Grover::Runtime::GroverVolume::getLsnManager() const at /local/p4clients/pkgbuild-fkx7d/workspace/src/GroverRuntime/./grover/runtime/grover-volume.h:1028)
/rdsdbbin/oscar/bin/mysqld() [0x19b1123] (in groverPartitionAllocateContiguousLsns at /local/p4clients/pkgbuild-fkx7d/workspace/src/GroverRuntime/grover/runtime/grover-csd.cpp:1458)
/rdsdbbin/oscar/bin/mysqld(_Z24log_allocate_grover_lsnsmPKmPP20_hgrover_write_statebb+0x153) [0x181a933]
/rdsdbbin/oscar/bin/mysqld(_Z35mtr_log_reserve_and_write_to_groverP5mtr_t+0xc1b) [0x182801b]
/rdsdbbin/oscar/bin/mysqld(_ZN5mtr_t6commitEv+0xcb) [0x16bddcb]
/rdsdbbin/oscar/bin/mysqld(_Z21dict_hdr_flush_row_idv+0xcb) [0x183b70b]
/rdsdbbin/oscar/bin/mysqld(_Z12row_ins_stepP9que_thr_t+0x5f7) [0x17284b7]
/rdsdbbin/oscar/bin/mysqld(_Z15que_run_threadsP9que_thr_t+0x4a7) [0x1702c07]
/rdsdbbin/oscar/bin/mysqld(_Z12que_eval_sqlP11pars_info_tPKcmP5trx_t+0x197) [0x1703657]
/rdsdbbin/oscar/bin/mysqld(_ZN8S3Stream20persistS3FileHistoryEmP5trx_t+0xe3) [0x1248ec3]
/rdsdbbin/oscar/bin/mysqld(_ZN8S3Stream22persistS3StreamHistoryEv+0x73) [0x1248ff3]
/rdsdbbin/oscar/bin/mysqld(_ZN8S3Stream4readEPhm+0x11f) [0x123b9bf]
/rdsdbbin/oscar/bin/mysqld(_my_b_s3_read+0xd3) [0x1156e33]
/rdsdbbin/oscar/bin/mysqld(_my_b_get+0x2f) [0x14b798f]
/rdsdbbin/oscar/bin/mysqld(_ZN9READ_INFO9next_lineEv+0x407) [0x126ef07]
/rdsdbbin/oscar/bin/mysqld(_Z10mysql_loadP3THDP12sql_exchangeP10TABLE_LISTR4ListI4ItemES8_S8_15enum_duplicates14LoadFileSource+0xa9f) [0x1272a3f]
/rdsdbbin/oscar/bin/mysqld(_Z30mysql_execute_command_internalP3THDb+0x1ba3) [0xfa5c43]
/rdsdbbin/oscar/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x12b) [0xfaa3ab]
/rdsdbbin/oscar/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3ab) [0xfab2eb]
/rdsdbbin/oscar/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1017) [0xfac497]
/rdsdbbin/oscar/bin/mysqld(_Z10do_commandP3THD+0x147) [0xfad107]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer7consumeEjjb+0x2a7) [0x10a15a7]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer5startEv+0x10f) [0x10a17af]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer11drain_queueEPv+0x4b) [0x10a19eb]
/rdsdbbin/oscar/bin/mysqld(pfs_spawn_thread+0xd7) [0x18dd437]
/lib64/libpthread.so.0(+0x72ab) [0x40001982a2ab]
/lib64/libc.so.6(+0xd5e5b) [0x4000212f2e5b]
03:14:59 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=9
max_threads=400
thread_count=9
connection_count=9
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 227952 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.


Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (401900cebae5): LOAD DATA FROM s3 's3://xxxxxx' INTO TABLE temp_table FIELDS TERMINATED BY '|' ENCLOSED BY '"' LINES TERMINATED BY ' ' IGNORE 1 LINES ( `id`)
Connection ID (thread ID): 18
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
aurora backtrace compare flag : 1
Writing a core file
2022-02-16T03:15:11.316356Z 0 [Warning] The syntax '--secure-auth' is deprecated and will be removed in a future release
2022-02-16T03:15:11.316576Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2022-02-16T03:15:11.316700Z 0 [Note] /rdsdbbin/oscar/bin/mysqld (mysqld 5.7.12) starting as process 19993 ...
2022-02-16T03:15:11.325755Z 0 [Warning] InnoDB: Setting innodb_checksums to OFF is DEPRECATED. This option may be removed in future releases. You should set innodb_checksum_algorithm=NONE instead.
2022-02-16T03:15:11.325795Z 0 [Note] InnoDB: Started in read only mode
2022-02-16T03:15:11.325803Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-02-16T03:15:11.325807Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-02-16T03:15:11.325810Z 0 [Note] InnoDB: Uses event mutexes
2022-02-16T03:15:11.325814Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-02-16T03:15:11.325817Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-02-16 03:15:11 0x400027042160[IB_LOG_LEVEL_INFO]:Initializing buffer pool, size = 89.7G (srv0start.cc:4406)
2022-02-16T03:15:11.325881Z 0 [Note] InnoDB: == Add SYNC_FAST DDL...
2022-02-16T03:15:11.326091Z 0 [Note] InnoDB: Number of pools: 1
2022-02-16T03:15:11.326217Z 0 [Note] InnoDB: Not using CPU crc32 instructions
2022-02-16T03:15:11.326224Z 0 [Note] InnoDB: Disabling background log and ibuf IO write threads.
2022-02-16T03:15:11.326944Z 0 [Note] InnoDB: Initializing buffer pool, total size = 89.7432G, instances = 16, chunk size = 5.60895G
2022-02-16 03:15:11 0x400027042160 Attempt to open data volume as READER
(buf0oscar.cc:799)
(buf0buf.cc:2411)
(buf0buf.cc:2411)
(buf0buf.cc:2411)
(buf0buf.cc:2411)
(buf0oscar.cc:799)
(buf0oscar.cc:799)
(buf0oscar.cc:799)
2022-02-16T03:15:13.952531Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-02-16T03:15:13.952807Z 0 [Note] InnoDB: page_cleaner coordinator priority: -20
2022-02-16T03:15:13.952815Z 0 [Note] InnoDB: page_cleaner worker priority: -20
2022-02-16T03:15:13.952825Z 0 [Note] InnoDB: page_cleaner worker priority: -20
2022-02-16T03:15:13.952825Z 0 [Note] InnoDB: page_cleaner worker priority: -20
2022-02-16T03:15:13.974660Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2022-02-16T03:15:14.009425Z 0 [Note] InnoDB: 5.7.12 started; log sequence number 0
2022-02-16T03:15:14.009612Z 0 [Note] Plugin 'FEDERATED' is disabled.
220216 3:15:14 server_audit: Audit STARTED.
Found DAS config file, trying to load DAS switcher from DAS config file.
2022-02-16 03:15:14 70369398759776:[DAS][INFO]: Calculated persistence threads 8
aurora_enable_das:0
220216 3:15:14 server_audit: server_audit_incl_users set to ''.
220216 3:15:14 server_audit: server_audit_excl_users set to ''.
2022-02-16T03:15:14.018244Z 0 [Warning] Read-only servers cannot create temp tables in InnoDB. Changing default engine for temp tables to MyISAM.
2022-02-16T03:15:14.018252Z 0 [Warning] Read-only servers cannot create intrinsic temp tables in InnoDB. Changing default engine for temp tables to MyISAM.
2022-02-16T03:15:14.019683Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
2022-02-16T03:15:14.020167Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2022-02-16T03:15:14.020229Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2022-02-16T03:15:14.020250Z 0 [Note] IPv6 is available.
2022-02-16T03:15:14.020258Z 0 [Note] - '::' resolves to '::';
2022-02-16T03:15:14.020373Z 0 [Note] Server socket created on IP: '::'.
2022-02-16T03:15:14.281071Z 0 [Note] /rdsdbbin/oscar/bin/mysqld: ready for connections.
Version: '5.7.12' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Server (GPL)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 11 100 11 0 0 11000 0 --:--:-- --:--:-- --:--:-- 11000
----------------------- END OF LOG ----------------------
asked 3 years ago661 views
2 Answers
0
Accepted Answer

This version of RDS has a bug with the query

LOAD DATA FROM S3

The AWS technicians contacted us and informed us that

The fix was use de load data of mysql, creating the file on runtime

answered 2 years ago
0

Hi

I understand you want to upgrade from Aurora 1.22.2 to 2.10.2 and you are getting a cycle of errors.Please correct me if I misunderstood.

Reviewing the logs you posted ,your cluster hits a bug .

I would recommend that you patch your cluster to the latest version to avoid future crashes. I would also recommend that if the issue persists open a case with AWS Support for further assistance [2].

I hope the was helpful!

Reference

[1]https://docs.aws.amazon.com/AmazonRDS/latest/AuroraMySQLReleaseNotes/AuroraMySQL.Updates.3010.html

[2]https://us-east-1.console.aws.amazon.com/support/home/?nc2=h_ql_cu&region=us-east-1#/

Karabo
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions