Moving 60M file from S3 to Local

0

We have a requirement of moving 60M files from S3 Bucket to Local and currently using move commands to perform this. But what we see is this it is bit slow.

Is there a way to faster this process?

Note: The individual files are not so huge but the no of files are more

2개 답변
0

Hello.

Is access to S3 a public line?
For example, if you use a dedicated line using something like DirectConnect, the line will be more stable and may be faster.
However, it is not realistic to set up a dedicated line just for downloading, so I think it is better to allow for a certain download speed.
https://repost.aws/knowledge-center/s3-troubleshoot-slow-downloads-uploads
https://repost.aws/knowledge-center/s3-download-slow-loading-web-browser

For example, it may be possible to download multiple files faster by combining them into several files.
You can use Python's S3 Concat to combine multiple files.
https://pypi.org/project/s3-concat/

I think it will be faster if you use the asyncio module in Python to perform parallel processing.

With AWS CLI, I think you can expect some speedup by setting "max_concurrent_requests" and "max_queue_size" large.
https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html

profile picture
전문가
답변함 2달 전
profile picture
전문가
검토됨 한 달 전
0

Hi, if your destination can be a NFS or SMB file share, I would recommend using AWS DataSync agent. Using DataSync agent, you can significantly speed up the small files transfer. However, you would need to setup an Agent either on a VM on-premises or on EC2. https://docs.aws.amazon.com/datasync/latest/userguide/working-with-locations.html

You may also look into using rclone : https://stackoverflow.com/a/71608811

psp
답변함 2달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠