AWS DMS Can't write to Redshift Intermediate Bucket

1

Hi,

Current setup:

I setup a DMS replication instance/endpoints and task from an RDS aurora mysql cluster to a redshift cluster

  • it's a full-load task for only one database + table (testing purpose)
  • the database is created and table "user_details" is filled with 10K rows

Current outcome:

  • the database is created on redshift cluster
  • user_details table schema is loaded and created correctly
  • user_details table **is empty **
  • task ended with Error and Table Error on "user_details"

Expected outcome diff:

  • user_details table should have 10K rows
  • task is successfull

Logs: I would expect data loaded from this lines in log:

2023-04-24T08:09:14 [SOURCE_UNLOAD   ]I:  Unload finished for table 'dbsource'.'user_details' (Id = 1). 10000 rows sent.  (streamcomponent.c:3543)
2023-04-24T08:09:14 [TARGET_LOAD     ]I:  Load finished for table 'dbsource'.'user_details' (Id = 1). 10000 rows received. 0 rows skipped. Volume transferred 6584376.  (streamcomponent.c:3838)

But also it seems redshift endpoint has hard time with S3 connection (but the log is not enough verbose to be sure)

2023-04-24T07:26:57 [FILE_FACTORY    ]E:  Write entire file failed: source = '/rdsdbdata/data/tasks/YYY/cloud/1/LOAD00000001.gzip' target = 'dms-zzz/redshift-XXX/YYY/1/LOAD00000001.csv' open type = 3 [1000731]  (at_universal_fs_object.c:316)

And also (still not verbose enough to undesrtand):

2023-04-24T07:20:30 [SOURCE_CAPTURE  ]W:  Got 0 headers at error exception  (anw_retry_strategy.cpp:91)
2023-04-24T07:20:30 [SOURCE_CAPTURE  ]E:  curlCode: 7, Couldn't connect to server [1001705]  (transfer_client.cpp:500)
2023-04-24T07:20:30 [SOURCE_CAPTURE  ]E:  Failed to upload file '/rdsdbdata/data/tasks/YYY/cloud/1/LOAD00000001.gzip' to bucket 'dms-zzz' as 'redshift-XXX/YYY/1/LOAD00000001.csv', status = 4 (FAILED) [1001705]  (transfer_client.cpp:514)
2023-04-24T07:20:30 [SOURCE_CAPTURE  ]E:  Failed to list bucket 'dms-zzz': error code <>: curlCode: 7, Couldn't connect to server [1001713]  (s3_dir_actions.cpp:105)

The default bucket is still created though, and AmazonDMSRedshiftS3Role policy is attached to the dms-access-for-endpoint IAM role has stated here https://repost.aws/knowledge-center/dms-redshift-connectivity-failures

Is there a way to know everything about the service/assumed role/request parameters for the "Couldn't connect to server" within the log ? This way I could check if it's a permission issue / network issue / etc...

Any help would be much appreciated

  • @rePost-User-4113219 Did you find a resolution here? We tried adding a VPC endpoint but are still experiencing the same error.

1 Answer
1

Did you create a VPC endpoint for S3 to be able to copy to so it can be loaded to Redshift. As of DMS 3.4.7 you need to use a VPC endpoint for Redshift as a target. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_VPC_Endpoints.html

AWS
Tony_N
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions