- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
What type of data are you trabferring, is it millions upon millions of little files or a lesser number of large files? If it's the former then disk seek times and access times will start to be a factor that slows things down.
You've probably seen this already, but AWS has recommendations to optimise the data transfer throughput https://docs.aws.amazon.com/snowball/latest/developer-guide/performance.html
When you copy files averaging less than 10MB in size to a device, you will experience slow performance. In this case, you have to package files into larger files (batching) and use parallelization (multiple copy threads at the same time). Please see this blog for more info: https://aws.amazon.com/blogs/storage/best-practices-for-accelerating-data-migrations-using-aws-snowball-edge/
I just found s5cmd which is helping a ton, upload speeds have increased exponentially. Do you have any experience with s5cmd? Would packaging/batching, along with s5cmd, increase the speed even more?
S5cmd is known to work well with the data set you described. When you have a mixed (very small and large files) you can batch the small files together and leave the large files as is. Please see snow-transfer-tool that automatically batches files during transfer to Snowball. https://github.com/aws-samples/snow-transfer-tool
I just learned of this tool not an hour ago, I'm looking into it. Thank you
Relevanter Inhalt
- AWS OFFICIALAktualisiert vor 3 Jahren
- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor 3 Jahren
Tons of raw photos. So not necessarily millions of little files and not necessarily a lesser number of large files.