How can I increase transfer/write speeds for Snowball migration?

Question

I have around 250 TB of data I am trying to move to AWS S3 so I ordered a Snowball.  I've attempted to upload to the Snowball via the AWS OpsHub and via the AWS CLI, both of which end up having around 8-9 mb/s upload speeds.... which is of course not going to cut it.  There just has to be a way to increase that speed and I feel I've exhausted my own resources.  I have tried uploading with the AWS OpsHub with 1gb internet and a 10 GB switch, I've tried mounting the Snowball as a local drive and using the CLI and all of them result in very slow write speeds.

Answer

When you copy files averaging less than 10MB in size to a device, you will experience slow performance. In this case, you have to package files into larger files (batching) and use parallelization (multiple copy threads at the same time). Please see this blog for more info: https://aws.amazon.com/blogs/storage/best-practices-for-accelerating-data-migrations-using-aws-snowball-edge/

Answer

What type of data are you trabferring, is it millions upon millions of little files or a lesser number of large files? If it's the former then disk seek times and access times will start to be a factor that slows things down.

You've probably seen this already, but AWS has recommendations to optimise the data transfer throughput https://docs.aws.amazon.com/snowball/latest/developer-guide/performance.html

Answer

S5cmd is known to work well with the data set you described. When you have a mixed (very small and large files) you can batch the small files together and leave the large files as is. Please see snow-transfer-tool that automatically batches files during transfer to Snowball. https://github.com/aws-samples/snow-transfer-tool

How can I increase transfer/write speeds for Snowball migration?

Relevant content