How do I troubleshoot low transfer speed on my Site-to-Site VPN?

Short Description

The following common network issues cause low transfer speed over a Site-to-Site VPN connection:

• Low performance routing
• Low maximum transmission units (MTU) along the path
• TCP/UDP throughput issues caused by a lower bandwidth tunnel
• Gateway device that throttles the data transfer
• Oversaturated VPN tunnel

The following common system or application issues also cause low transfer speed:

• Application endpoint resource quotas
• Application endpoints that use a single TCP stream for the data transfer

Resolution

To troubleshoot low transfer speeds on Site-to-Site VPN, take the following actions.

Check for low performance routing

First, install the Linux MTR tool. Most Linux distributions come with MTR preinstalled. You can also download the tool from your distribution's software package manager:

• To install MTR for Amazon Linux, run the following command:

  sudo yum install mtr

• To install MTR for Ubuntu, run the following command:

  sudo apt-get install mtr-tiny

Then, use MTR from your on-premises endpoint to run the following command:

mtr -T -nrz example_IP

Note: Replace example_IP with the public IP address of your AWS tunnel.

The MTR output includes the Autonomous System Number (ASN) of the internet service provider (ISP) along the path. Review the output to verify that the traffic flows along a direct path from the customer gateway device to AWS.

If needed, use the MTR output to identify latency over the internet.

Check the MTU along the path

Identify the lowest MTU in your path:

• For Linux, run the following command:

  ping  example_IP -M do -s 1460

• For Windows, run the following command:

  ping  example_IP -l 1460 -f

Note: Replace example_IP with your destination's IP address and 1460 with your Maximum Segment Size (MSS) in bytes.

If you receive a "packet needs to be fragmented but DF set" message, then lower the MSS size by 1, and run the command again. Repeat the test until the output shows a successful ping.

Troubleshoot TCP/UDP throughput issues

Prerequisites: You must install the iPerf3 tool. To install iPerf3, see Download iPerf binaries on the iPerf website.

Test the UDP bandwidth capabilities from the server and the client:

• From the server, run the following command:

  sudo iperf -s -u

• From the client, run the following command:

  sudo iperf3 -i 1 -u -p 33344 -b 1.2G -c example_IP -V

Note: Replace example_IP with the server's private IP address.

Make sure that your bandwidth credit is available for the Amazon Elastic Compute Cloud (Amazon EC2) instance associated with the VPN. If the bandwidth credit isn't available, then create a larger instance size, and then test again.

Test the TCP throughput from the server and the client.

• From the server, run the following command:

  iperf3 -s

• From the client, run the following command:

  sudo iperf3 -c example_IP -P 10 -w 128K -V
  sudo iperf3 -c example_IP -P 10 -w 512K -V
  sudo iperf3 -c example_IP -P 10 -w 1024K -V   

Note: Replace example_IP with the server's private IP address.

To test the source and destination memory buffers when you increase the instance size, test different TCP receive window sizes.

Make sure that the customer gateway device allows data transfer

Take a packet capture (PCAP) on the customer gateway device. If the PCAP shows high latency or packet drops on the customer gateway device, then check the device configuration settings. Use this information to verify whether the customer gateway is throttling the data transfer. To troubleshoot issues, check the vendor documentation for your device, or contact vendor support for the device.

Verify that the VPN tunnel isn't oversaturated

The maximum bandwidth for each Site-to-Site VPN tunnel is 1.25 gigabits per second (Gbps). To verify that the throughput of your tunnels doesn't exceed the quota, complete the following steps:

1. Open the Amazon CloudWatch console.
2. Choose Site-to-Site VPN metrics.
3. Select TunnelDataIn and TunnelDataOut.
4. For Statistic, select Sum, and then for Period, select 5 minutes.
5. To calculate your tunnel's throughput in Gbps, use the following equation:
   (((m1+m2)/300)*8)/1,000,000,000
   Note: In this equation, replace m1 with the highest value of TunnelDataIn, and m2 with the highest value of TunnelDataOut.
6. Check whether the throughput is greater than 1.2 Gbps. For throughput greater than 1.2 Gbps, launch two Border Gateway Protocol (BGP) tunnels with equal-cost multi-path routing (ECMP), and an AWS Transit Gateway transit gateway.

Check whether the application endpoints exceed their resource quotas

On the customer gateway device, use Wireshark (for Windows) or tcdump (for Linux) to take a PCAP when data flows through the VPN tunnel.

Important: Start the PCAP before you initiate the data transfer. It's a best practice to take the PCAP simultaneously on both the source and destination end hosts, and on the on-premises customer gateway.

Review the PCAP output for delays and packet retransmissions. If you find unexpected latency or packet retransmission on the end hosts, then contact the customer gateway device's vendor support.

Test for additional TCP streams

Complete the following steps:

1. Test the number of TCP streams that the application uses.
   From the server, run the following command:

   iperf3 -s

   From the client, run the following command:

   sudo iperf3 -c example_IP -P 10 -t 30

   Note: Replace example_IP with the server's private IP address. The preceding client command tests 10 parallel TCP streams.
2. To verify whether more parallel TCP streams boost the data transfer, repeat the test with more TCP streams. Example:

   sudo iperf3 -c example_IP -P 30 -t 30

   Note: The preceding command tests 30 TCP streams.
3. If the throughput increases when you add streams to the test, then contact the software vendor support to add TCP streams to the application.

Related information

Enhanced networking on Amazon EC2 instances

AWS VPN FAQs