Improving copying speed across EC2 instances

0

Dear community,

Background:

We are working on a project where we have to transfer .db files from one ec2 instance to another more frequently. Both the instances are r5.2xlarge. The .db file is around 100 GB in size. We are using the Bastion host to access the actual EC2 instances. We are trying to copy a file from the bastion host because the actual EC2 instances cannot connect to each other.

Issue:

When we try to copy the file using scp command, the first 16GB is normally fast, but it slows down very much after that, taking a huge amount of time to complete the transfer. We have tried many times and this issue is reproducible.

***Command: ***scp -3 ec2-user@ xx.x.xx.xxx:source file address ec2-user@xx.x.xx.xxx:destination file address

The scp -3 used in the command is to avoid the actual files copying via Bastion host.

Expectation:

Can you please provide us some suggestions or recommendations to improve the copying speed across these instances?

Thanks in advance.

asked 2 years ago1154 views
3 Answers
1

Looking at the instance type documentation the r5.2xlarge has "up to 10 Gb/s" of networking and there is a footnote that says "Instances marked with "Up to" Network Bandwidth have a baseline bandwidth and can use a network I/O credit mechanism to burst beyond their baseline bandwidth on a best effort basis."

That indicates to me that you're getting some burst bandwidth at the beginning of the copy operation but then running out of credits so the speed is reduced to the baseline bandwidth.

I'd recommend choosing an instance with a higher network performance rating.

profile pictureAWS
EXPERT
answered 2 years ago
1

I would love to add something more to the fantastic answer provided by Brettski@AWS. As per AWS documentation, the network bandwidth available to an EC2 instance depends on the Destination of the Traffic. So if it's Within the Region, then the traffic can utilize the full network bandwidth available to the instance, but to other regions, an internet gateway, or Direct Connect, traffic can utilize up to 50% of the network bandwidth available to a current generation instance with a minimum of 32 vCPUs. Bandwidth for a current-generation instance with less than 32 vCPUs is limited to 5 Gbps.

And lastly adding more to your question, instances with 16 vCPUs or fewer (size 4xlarge and smaller) are documented as having "up to" a specified bandwidth; for example, "up to 10 Gbps".

These instances have a baseline bandwidth, however, to meet additional demand, they can use a network I/O credit mechanism to burst beyond their baseline bandwidth for a limited time, typically from 5 to 60 minutes, depending on the instance size. And this is the reason that an instance receives the maximum number of network I/O credits at launch.

And once the instance exhausts its network I/O credits, it returns to its baseline bandwidth.

So if you compare this with your scenario you will find, you too getting a good copying speed at the beginning but later on it reduces as the instance exhausts its network I/O credits, forcing it to return to baseline bandwidth.

You can refer to more from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html

answered 2 years ago
  • Thank you so much for the detailed answer and providing the links.

0

While as you pointed out the cause of the issue and suggested to upgrade the instance to get a fixed bandwidth, we tried with using S3 as the intermediate option between two EC2 instances and it worked out pretty well. We will also try to upgrade the instance and see the improvement. Thanks.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions