Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Improving copying speed across EC2 instances

0

Dear community,

Background:

We are working on a project where we have to transfer .db files from one ec2 instance to another more frequently. Both the instances are r5.2xlarge. The .db file is around 100 GB in size. We are using the Bastion host to access the actual EC2 instances. We are trying to copy a file from the bastion host because the actual EC2 instances cannot connect to each other.

Issue:

When we try to copy the file using scp command, the first 16GB is normally fast, but it slows down very much after that, taking a huge amount of time to complete the transfer. We have tried many times and this issue is reproducible.

***Command: ***scp -3 ec2-user@ xx.x.xx.xxx:source file address ec2-user@xx.x.xx.xxx:destination file address

The scp -3 used in the command is to avoid the actual files copying via Bastion host.

Expectation:

Can you please provide us some suggestions or recommendations to improve the copying speed across these instances?

Thanks in advance.

Topics
Compute
Tags
Amazon EC2ComputeFile Sharing
Language
English
asked 4 years ago1.7K views
2 Answers
1

Looking at the instance type documentation the r5.2xlarge has "up to 10 Gb/s" of networking and there is a footnote that says "Instances marked with "Up to" Network Bandwidth have a baseline bandwidth and can use a network I/O credit mechanism to burst beyond their baseline bandwidth on a best effort basis."

That indicates to me that you're getting some burst bandwidth at the beginning of the copy operation but then running out of credits so the speed is reduced to the baseline bandwidth.

I'd recommend choosing an instance with a higher network performance rating.

AWS
EXPERT
answered 4 years ago
0

While as you pointed out the cause of the issue and suggested to upgrade the instance to get a fixed bandwidth, we tried with using S3 as the intermediate option between two EC2 instances and it worked out pretty well. We will also try to upgrade the instance and see the improvement. Thanks.

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content