AWS DMS Can't write to Redshift Intermediate Bucket

1

Hi,

Current setup:

I setup a DMS replication instance/endpoints and task from an RDS aurora mysql cluster to a redshift cluster

  • it's a full-load task for only one database + table (testing purpose)
  • the database is created and table "user_details" is filled with 10K rows

Current outcome:

  • the database is created on redshift cluster
  • user_details table schema is loaded and created correctly
  • user_details table **is empty **
  • task ended with Error and Table Error on "user_details"

Expected outcome diff:

  • user_details table should have 10K rows
  • task is successfull

Logs: I would expect data loaded from this lines in log:

2023-04-24T08:09:14 [SOURCE_UNLOAD   ]I:  Unload finished for table 'dbsource'.'user_details' (Id = 1). 10000 rows sent.  (streamcomponent.c:3543)
2023-04-24T08:09:14 [TARGET_LOAD     ]I:  Load finished for table 'dbsource'.'user_details' (Id = 1). 10000 rows received. 0 rows skipped. Volume transferred 6584376.  (streamcomponent.c:3838)

But also it seems redshift endpoint has hard time with S3 connection (but the log is not enough verbose to be sure)

2023-04-24T07:26:57 [FILE_FACTORY    ]E:  Write entire file failed: source = '/rdsdbdata/data/tasks/YYY/cloud/1/LOAD00000001.gzip' target = 'dms-zzz/redshift-XXX/YYY/1/LOAD00000001.csv' open type = 3 [1000731]  (at_universal_fs_object.c:316)

And also (still not verbose enough to undesrtand):

2023-04-24T07:20:30 [SOURCE_CAPTURE  ]W:  Got 0 headers at error exception  (anw_retry_strategy.cpp:91)
2023-04-24T07:20:30 [SOURCE_CAPTURE  ]E:  curlCode: 7, Couldn't connect to server [1001705]  (transfer_client.cpp:500)
2023-04-24T07:20:30 [SOURCE_CAPTURE  ]E:  Failed to upload file '/rdsdbdata/data/tasks/YYY/cloud/1/LOAD00000001.gzip' to bucket 'dms-zzz' as 'redshift-XXX/YYY/1/LOAD00000001.csv', status = 4 (FAILED) [1001705]  (transfer_client.cpp:514)
2023-04-24T07:20:30 [SOURCE_CAPTURE  ]E:  Failed to list bucket 'dms-zzz': error code <>: curlCode: 7, Couldn't connect to server [1001713]  (s3_dir_actions.cpp:105)

The default bucket is still created though, and AmazonDMSRedshiftS3Role policy is attached to the dms-access-for-endpoint IAM role has stated here https://repost.aws/knowledge-center/dms-redshift-connectivity-failures

Is there a way to know everything about the service/assumed role/request parameters for the "Couldn't connect to server" within the log ? This way I could check if it's a permission issue / network issue / etc...

Any help would be much appreciated

  • @rePost-User-4113219 Did you find a resolution here? We tried adding a VPC endpoint but are still experiencing the same error.

1 Antwort
1

Did you create a VPC endpoint for S3 to be able to copy to so it can be loaded to Redshift. As of DMS 3.4.7 you need to use a VPC endpoint for Redshift as a target. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_VPC_Endpoints.html

AWS
Tony_N
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen