Transfer server - bulk import - rsync

0

Hello, I'd like to ask what is the recommended way for bulk import of files into the AWS Transfer service.
We'd like to migrate all files from one of our current SFTP servers into the AWS S3 bucket using Transfer server and use that transfer server for future sftp uploads. What would you recommend to use for such use case.
I'm not sure if it's possible to use tools like rsync, since it seems that rsync needs remote ssh shell on the sftp server to do that, but AWS Transfer doesn't seem to allow that.
Is it possible to utilize rsync here or is there any other way?
Thanks

petergh
preguntada hace 5 años479 visualizaciones
4 Respuestas
0

Hi petergh,
One way you could migrate files from your on-prem storage server (NFS) into your S3 buckets is by using AWS DataSync (https://aws.amazon.com/datasync/). Once you've seeded your bucket with data, you can use the same bucket(s) for access through AWS Transfer for SFTP.

Please let me know if you have any questions about this set up or run into issues.

Thanks,
Smitha

AWS
respondido hace 5 años
0

Proposed solution with AWS DataSync seems reasonable, however we didnt choose it as a way to go. We were also thinking about using simple aws s3 sync command.
However in the end we will be transferring 10+ TB of data so we might end up with the AWS Snowball service. Thanks

petergh
respondido hace 5 años
0

petergh:

Can you share your pros/cons of using Snowball over DataSync?

Thanks
Paul

AWS
Paul_R
respondido hace 5 años
0

I guess most significant reason for snowball was the bandwith.
We couldnt consume all of 1Gbps bandwith just for this transfer and with limiting bandwith it would take at least 3 weeks to transfer 10+ TB and this is with the transfer running non-stop the entire time.
With the snowball this can be achieved in less time even with the shipping times included.

petergh
respondido hace 5 años

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas