Transfer server - bulk import - rsync

Question

Hello, I'd like to ask what is the recommended way for bulk import of files into the AWS Transfer service.  
We'd like to migrate all files from one of our current SFTP servers into the AWS S3 bucket using Transfer server and use that transfer server for future sftp uploads. What would you recommend to use for such use case.  
I'm not sure if it's possible to use tools like rsync, since it seems that rsync needs remote ssh shell on the sftp server to do that, but AWS Transfer doesn't seem to allow that.  
Is it possible to utilize rsync here or is there any other way?  
Thanks

petergh · Answer

I guess most significant reason for snowball was the bandwith.   
We couldnt consume all of 1Gbps bandwith just for this transfer and with limiting bandwith it would take at least 3 weeks to transfer 10+ TB and this is with the transfer running non-stop the entire time.  
With the snowball this can be achieved in less time even with the shipping times included.

petergh · Answer

Proposed solution with AWS DataSync seems reasonable, however we didnt choose it as a way to go. We were also thinking about using simple aws s3 sync command.  
However in the end we will be transferring 10+ TB of data so we might end up with the AWS Snowball service. Thanks

AWS-User-9826533 · Answer

Hi petergh,  
One way you could migrate files from your on-prem storage server (NFS) into your S3 buckets is by using AWS DataSync (https://aws.amazon.com/datasync/). Once you've seeded your bucket with data, you can use the same bucket(s) for access through AWS Transfer for SFTP.  
  
Please let me know if you have any questions about this set up or run into issues.  
  
Thanks,  
Smitha

Paul_R · Answer

petergh:  
  
Can you share your pros/cons of using Snowball over DataSync?  
  
Thanks  
Paul

Transfer server - bulk import - rsync

Relevant content