Moving 60M file from S3 to Local

0

We have a requirement of moving 60M files from S3 Bucket to Local and currently using move commands to perform this. But what we see is this it is bit slow.

Is there a way to faster this process?

Note: The individual files are not so huge but the no of files are more

已提问 2 个月前196 查看次数
2 回答
0

Hello.

Is access to S3 a public line?
For example, if you use a dedicated line using something like DirectConnect, the line will be more stable and may be faster.
However, it is not realistic to set up a dedicated line just for downloading, so I think it is better to allow for a certain download speed.
https://repost.aws/knowledge-center/s3-troubleshoot-slow-downloads-uploads
https://repost.aws/knowledge-center/s3-download-slow-loading-web-browser

For example, it may be possible to download multiple files faster by combining them into several files.
You can use Python's S3 Concat to combine multiple files.
https://pypi.org/project/s3-concat/

I think it will be faster if you use the asyncio module in Python to perform parallel processing.

With AWS CLI, I think you can expect some speedup by setting "max_concurrent_requests" and "max_queue_size" large.
https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html

profile picture
专家
已回答 2 个月前
profile picture
专家
已审核 1 个月前
0

Hi, if your destination can be a NFS or SMB file share, I would recommend using AWS DataSync agent. Using DataSync agent, you can significantly speed up the small files transfer. However, you would need to setup an Agent either on a VM on-premises or on EC2. https://docs.aws.amazon.com/datasync/latest/userguide/working-with-locations.html

You may also look into using rclone : https://stackoverflow.com/a/71608811

psp
已回答 2 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则