- 最新
- 最多得票
- 最多評論
Hello.
Is access to S3 a public line?
For example, if you use a dedicated line using something like DirectConnect, the line will be more stable and may be faster.
However, it is not realistic to set up a dedicated line just for downloading, so I think it is better to allow for a certain download speed.
https://repost.aws/knowledge-center/s3-troubleshoot-slow-downloads-uploads
https://repost.aws/knowledge-center/s3-download-slow-loading-web-browser
For example, it may be possible to download multiple files faster by combining them into several files.
You can use Python's S3 Concat to combine multiple files.
https://pypi.org/project/s3-concat/
I think it will be faster if you use the asyncio module in Python to perform parallel processing.
With AWS CLI, I think you can expect some speedup by setting "max_concurrent_requests" and "max_queue_size" large.
https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html
Hi, if your destination can be a NFS or SMB file share, I would recommend using AWS DataSync agent. Using DataSync agent, you can significantly speed up the small files transfer. However, you would need to setup an Agent either on a VM on-premises or on EC2. https://docs.aws.amazon.com/datasync/latest/userguide/working-with-locations.html
You may also look into using rclone : https://stackoverflow.com/a/71608811
相關內容
- 已提問 1 天前
- 已提問 1 年前
- AWS 官方已更新 2 年前
- AWS 官方已更新 1 年前
- AWS 官方已更新 2 年前
- AWS 官方已更新 1 年前