DMS task stuck

Question

Hi,  
I have a problem with a DMS task connecting an Aurora instance (source) to a redshift instance (target).  
Apparently the task is stuck at 97% while migrating some big tables (10 million rows each approx).  
We recently migrated from MySql to Aurora, hence the need to rerun the full migration.  
The task is "Full load, ongoing replication".  
  
After 8 hours of migration going on, the task restarts with an error:  
  
019-07-30T10:57:35 \[SOURCE_UNLOAD ]E: Connection error \[1022506] (mysql_endpoint_imp.c:106)  
2019-07-30T10:57:35 \[SOURCE_UNLOAD ]E: Endpoint is disconnected \[1020414] (endpointshell.c:2766)  
2019-07-30T10:57:35 \[SOURCE_UNLOAD ]E: Error executing source loop \[1020414] (streamcomponent.c:1596)  
2019-07-30T10:57:35 \[TASK_MANAGER ]E: Stream component failed at subtask 1, component st_1_J7ZYBQT4N6DCQRNGEDCD646RRA \[1020414] (subtask.c:1350)  
2019-07-30T10:57:35 \[SOURCE_UNLOAD ]E: Stream component 'st_1_J7ZYBQT4N6DCQRNGEDCD646RRA' terminated \[1020414] (subtask.c:1513)  
2019-07-30T10:57:35 \[TASK_MANAGER ]E: Task error notification received from subtask 1, thread 0 \[1020414] (replicationtask.c:2379)  
2019-07-30T11:01:47 \[SOURCE_UNLOAD ]E: Connection error \[1022506] (mysql_endpoint_imp.c:106)  
2019-07-30T11:01:47 \[SOURCE_UNLOAD ]E: Endpoint is disconnected \[1020414] (endpointshell.c:2766)  
2019-07-30T11:01:47 \[SOURCE_UNLOAD ]E: Error executing source loop \[1020414] (streamcomponent.c:1596)  
2019-07-30T11:01:47 \[TASK_MANAGER ]E: Stream component failed at subtask 2, component st_2_J7ZYBQT4N6DCQRNGEDCD646RRA \[1020414] (subtask.c:1350)  
2019-07-30T11:01:47 \[SOURCE_UNLOAD ]E: Stream component 'st_2_J7ZYBQT4N6DCQRNGEDCD646RRA' terminated \[1020414] (subtask.c:1513)  
2019-07-30T11:16:02 \[SOURCE_UNLOAD ]E: Connection error \[1022506] (mysql_endpoint_imp.c:106)  
2019-07-30T11:16:02 \[SOURCE_UNLOAD ]E: Endpoint is disconnected \[1020414] (endpointshell.c:2766)  
2019-07-30T11:16:02 \[SOURCE_UNLOAD ]E: Error executing source loop \[1020414] (streamcomponent.c:1596)  
2019-07-30T11:16:02 \[TASK_MANAGER ]E: Stream component failed at subtask 3, component st_3_J7ZYBQT4N6DCQRNGEDCD646RRA \[1020414] (subtask.c:1350)  
2019-07-30T11:16:02 \[SOURCE_UNLOAD ]E: Stream component 'st_3_J7ZYBQT4N6DCQRNGEDCD646RRA' terminated \[1020414] (subtask.c:1513)  
2019-07-30T11:19:24 \[SOURCE_UNLOAD ]E: Connection error \[1022506] (mysql_endpoint_imp.c:106)  
2019-07-30T11:19:24 \[SOURCE_UNLOAD ]E: Endpoint is disconnected \[1020414] (endpointshell.c:2766)  
2019-07-30T11:19:24 \[SOURCE_UNLOAD ]E: Error executing source loop \[1020414] (streamcomponent.c:1596)  
2019-07-30T11:19:24 \[TASK_MANAGER ]E: Stream component failed at subtask 4, component st_4_J7ZYBQT4N6DCQRNGEDCD646RRA \[1020414] (subtask.c:1350)  
2019-07-30T11:19:24 \[SOURCE_UNLOAD ]E: Stream component 'st_4_J7ZYBQT4N6DCQRNGEDCD646RRA' terminated \[1020414] (subtask.c:1513)  
  
This has happened at least twice since we started the task.  
  
regards,  
Matteo

Answer

I solved it by setting some endpoint settings(!) on the redshift end of things:  
  
maxFileSize=1048576;fileTransferUploadStreams=64  
  
This prevented the timeout from happening again and permitted the completion of the process.  
  
regards,,  
Matteo

DMS task stuck

Relevant content