Moving 60M file from S3 to Local

0

We have a requirement of moving 60M files from S3 Bucket to Local and currently using move commands to perform this. But what we see is this it is bit slow.

Is there a way to faster this process?

Note: The individual files are not so huge but the no of files are more

2 Answers
0

Hello.

Is access to S3 a public line?
For example, if you use a dedicated line using something like DirectConnect, the line will be more stable and may be faster.
However, it is not realistic to set up a dedicated line just for downloading, so I think it is better to allow for a certain download speed.
https://repost.aws/knowledge-center/s3-troubleshoot-slow-downloads-uploads
https://repost.aws/knowledge-center/s3-download-slow-loading-web-browser

For example, it may be possible to download multiple files faster by combining them into several files.
You can use Python's S3 Concat to combine multiple files.
https://pypi.org/project/s3-concat/

I think it will be faster if you use the asyncio module in Python to perform parallel processing.

With AWS CLI, I think you can expect some speedup by setting "max_concurrent_requests" and "max_queue_size" large.
https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html

profile picture
EXPERT
answered 2 months ago
profile picture
EXPERT
reviewed a month ago
0

Hi, if your destination can be a NFS or SMB file share, I would recommend using AWS DataSync agent. Using DataSync agent, you can significantly speed up the small files transfer. However, you would need to setup an Agent either on a VM on-premises or on EC2. https://docs.aws.amazon.com/datasync/latest/userguide/working-with-locations.html

You may also look into using rclone : https://stackoverflow.com/a/71608811

psp
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions