Moving 60M file from S3 to Local

0

We have a requirement of moving 60M files from S3 Bucket to Local and currently using move commands to perform this. But what we see is this it is bit slow.

Is there a way to faster this process?

Note: The individual files are not so huge but the no of files are more

2 Antworten
0

Hello.

Is access to S3 a public line?
For example, if you use a dedicated line using something like DirectConnect, the line will be more stable and may be faster.
However, it is not realistic to set up a dedicated line just for downloading, so I think it is better to allow for a certain download speed.
https://repost.aws/knowledge-center/s3-troubleshoot-slow-downloads-uploads
https://repost.aws/knowledge-center/s3-download-slow-loading-web-browser

For example, it may be possible to download multiple files faster by combining them into several files.
You can use Python's S3 Concat to combine multiple files.
https://pypi.org/project/s3-concat/

I think it will be faster if you use the asyncio module in Python to perform parallel processing.

With AWS CLI, I think you can expect some speedup by setting "max_concurrent_requests" and "max_queue_size" large.
https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html

profile picture
EXPERTE
beantwortet vor 2 Monaten
profile picture
EXPERTE
überprüft vor einem Monat
0

Hi, if your destination can be a NFS or SMB file share, I would recommend using AWS DataSync agent. Using DataSync agent, you can significantly speed up the small files transfer. However, you would need to setup an Agent either on a VM on-premises or on EC2. https://docs.aws.amazon.com/datasync/latest/userguide/working-with-locations.html

You may also look into using rclone : https://stackoverflow.com/a/71608811

psp
beantwortet vor 2 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen