Transfer server - bulk import - rsync

0

Hello, I'd like to ask what is the recommended way for bulk import of files into the AWS Transfer service.
We'd like to migrate all files from one of our current SFTP servers into the AWS S3 bucket using Transfer server and use that transfer server for future sftp uploads. What would you recommend to use for such use case.
I'm not sure if it's possible to use tools like rsync, since it seems that rsync needs remote ssh shell on the sftp server to do that, but AWS Transfer doesn't seem to allow that.
Is it possible to utilize rsync here or is there any other way?
Thanks

petergh
asked 5 years ago459 views
4 Answers
0

Hi petergh,
One way you could migrate files from your on-prem storage server (NFS) into your S3 buckets is by using AWS DataSync (https://aws.amazon.com/datasync/). Once you've seeded your bucket with data, you can use the same bucket(s) for access through AWS Transfer for SFTP.

Please let me know if you have any questions about this set up or run into issues.

Thanks,
Smitha

AWS
answered 5 years ago
0

Proposed solution with AWS DataSync seems reasonable, however we didnt choose it as a way to go. We were also thinking about using simple aws s3 sync command.
However in the end we will be transferring 10+ TB of data so we might end up with the AWS Snowball service. Thanks

petergh
answered 5 years ago
0

petergh:

Can you share your pros/cons of using Snowball over DataSync?

Thanks
Paul

AWS
Paul_R
answered 5 years ago
0

I guess most significant reason for snowball was the bandwith.
We couldnt consume all of 1Gbps bandwith just for this transfer and with limiting bandwith it would take at least 3 weeks to transfer 10+ TB and this is with the transfer running non-stop the entire time.
With the snowball this can be achieved in less time even with the shipping times included.

petergh
answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions