low s3 speed from EC2

Question

Hello.
I have a 1TB tar file that I am trying to upload to s3 from mg8.large instance type from us-east-1. I have enabled the s3 gateway VPC endpoint. 
I am using aws cli to upload. I have two questions:
1. The speed I am getting peaks at 33MiB/s. What can I do to increase the speed? 
2. How can I confirm that the upload uses the VPC endpoint?

Chetan Agrawal · Answer

Hi,

For #1 you can refer to this post: https://repost.aws/knowledge-center/s3-transfer-data-bucket-instance
For #2 check the route table for the subnet in which your EC2 instance is hosted. Route table should contain the entry for S3 traffic through the VPC gateway endpoint.

Regards,
Chetan

Didier Durand · Answer

Hi,

When using CLI for S3 transfers, you can optimize some parameters like `max-concurrent-requests`. See https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html#max-concurrent-requests

Also have a look at other config params on same page:

```
max_concurrent_requests - The maximum number of concurrent requests.
max_queue_size - The maximum number of tasks in the task queue.
multipart_threshold - The size threshold the CLI uses for multipart transfers of individual files.
multipart_chunksize - When using multipart transfers, this is the chunk size that the CLI uses for 
        multipart transfers of individual files.
max_bandwidth - The maximum bandwidth that will be consumed for uploading and downloading 
       data to and from Amazon S3.
```

On your second question, just traceroute your traffic from our EC2 instance by logging onto it via ssh and confirm that the IP address used for the s3 transfer matches the ip address of the dns of the endpoint

Best,

Didier

low s3 speed from EC2

Add your answer

Relevant content