How can I troubleshoot Direct Connect network performance issues?

Resolution

Note: It's a best practice to set up an on-premises dedicated test machine with an Amazon Virtual Private Cloud (Amazon VPC). Use Amazon Elastic Compute Cloud (Amazon EC2) instance type size C5 or larger.

Check for network or application issues

Use the iPerf3 tool to measure network bandwidth and compare the results with other applications or tools. For more information, see What is iPerf / iPerf3? on the iPerf website.

Complete the following steps:

1. To install iPerf3, run one of the following commands for the operating system (OS) that you use.

   Linux and Red Hat Enterprise Linux (RHEL):

   sudo yum install iperf3 -y

   Ubuntu:

   sudo apt install iperf3 -y

2. To bidirectionally measure the throughput, run the following iPerf3 commands for your server and client:

   Amazon EC2 instance (server):

   iperf3 -s -V

   On-premises localhost (client):

   iperf3 -c <private IP of EC2> -P 15 -t 15
   iperf3 -c <private IP of EC2> -P 15 -t 15 -R
   
   iperf3 -c <private IP of EC2> -w 256K
   iperf3 -c <private IP of EC2> -w 256K -R
   
   iperf3 -c <private IP of EC2> -u -b 1G -t 15
   iperf3 -c <private IP of EC2> -u -b 1G -t 15 -R
   

   Note: -P sets parallel threads for maximum throughput. -R reverses test direction, and -u uses User Datagram Protocol (UDP) instead of TCP to measure packet loss.

In the following example TCP test results, Bitrate is the measured throughput or transmission speed. Transfer is the total amount of data that's exchanged between client and server. Retry is the number of retransmitted packets. Retransmission happens on the sender side:

[ ID] Interval Transfer Bitrate Retry [SUM] 0.00-15.00 sec 7.54 GBytes 4.32 Gbits/sec 18112 sender [SUM] 0.00-15.00 sec 7.52 GBytes 4.31 Gbits/sec receiver

In the following example UDP test results, Lost is 0% on the sender side. This is because the sender (ID) and receiver (5) pair sent the maximum number of UDP datagrams. Lost/Total datagrams shows how many packets the receiver lost and the loss rate. In this example, the receiver lost 79% of network traffic:

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-15.00 sec 8.22 GBytes 4.71 Gbits/sec 0.000 ms 0/986756 (0%) sender [ 5] 0.00-15.00 sec 1.73 GBytes 989 Mbits/sec 0.106 ms 779454/986689 (79%) receiver

Note: If your Direct Connect connection uses AWS Site-to-Site VPN over a public virtual interface, then run performance tests without the VPN.

View metrics and interface counters

Use Amazon CloudWatch Logs to view metrics that can help you troubleshoot. For example, view ConnectionErrorCount and look for non-zero values that show a Media Access Control (MAC) error on an AWS device. You can use the Sum statistic for the ConnectionErrorCount metric.

View ConnectionLightLevelTx and ConnectionLightLevelRx to check whether the optical signal readings are within the range of -14.4 and 2.50 dBm.

View ConnectionBpsEgress, ConnectionBpsIngress, VirtualInterfaceBpsEgress, and VirtualInterfaceBpsIngress to make sure that the bitrate didn't reach the maximum bandwidth.

For more information, see Direct Connect metrics and dimensions.

If you use a hosted virtual interface that shares the total bandwidth with other users, then ask the Direct Connect owner about connection utilization. For more information, see Hybrid network connections.

Check your router and firewall at the Direct Connect location for the following issues:

• Look for irregularities in CPU, memory, port utilization, drops, and discards.
• Run the show interfaces statistics command to identify interface input and output errors, such as Cyclic Redundancy Check (CRC), frame, collisions, and carrier errors.
• For increased interface errors, clean or replace the fiber patch lead and Small Form-factor Pluggable (SFP) module.

Also, check whether Direct Connect is scheduled for maintenance.

Bidirectionally run MTR to check the network path

Run the My Traceroute (MTR) command for Linux to analyze network performance. For Windows, it's a best practice to turn on WSL 2 so that you can install MTR on a Linux subsystem. For more information, see What's new in WSL 2 on the Microsoft website. You can download WinMTR from the SourceForge website.

To bidirectionally run MTR, complete the following steps:

1. To install MTR, run one of the following commands for your OS:

   Amazon Linux/RHEL:

   sudo yum install mtr -y

   Ubuntu:

   sudo apt install mtr -y

2. For the on-premises to AWS direction, run MTR on the localhost (ICMP and TCP based):

   mtr -n -c 100 <private IP of EC2> --report$ mtr -n -T -P <EC2 instance open TCP port> -c 100 <private IP of EC2> --report

3. For the AWS to on-premises direction, run MTR on the EC2 instance (ICMP and TCP based):

   mtr -n -c 100 <private IP of the local host> --report$ mtr -n -T -P <local host open TCP port> -c 100 <private IP of the local host> --report

The following example command and output shows ICMP-based MTR results:

mtr -n -c 100 192.168.52.10 --report
Start: Sat Oct 30 20:54:39 2021
HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.101.222               0.0%   100    0.7   0.7   0.6   0.9   0.0
  2.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.110.120.2               0.0%   100  266.5 267.4 266.4 321.0   4.8
  4.|-- 10.110.120.1              54.5%   100  357.6 383.0 353.4 423.7  19.6
  5.|-- 192.168.52.10             47.5%   100  359.4 381.3 352.4 427.9  20.6

The following example command and output shows TCP-based MTR results:

mtr -n -T -P 80 -c 100 192.168.52.10 --report
Start: Sat Oct 30 21:03:48 2021
HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.101.222               0.0%   100    0.9   0.7   0.7   1.1   0.0
  2.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.110.120.2               0.0%   100  264.1 265.8 263.9 295.3   3.4
  4.|-- 10.110.120.1               8.0%   100  374.3 905.3 354.4 7428. 1210.6
  5.|-- 192.168.52.10             12.0%   100  400.9 1139. 400.4 7624. 1384.3

In the preceding results, each line in a hop represents a network device for a data packet that passes from the source to the destination. For more information about how to read MTR test results, see Reading MTR output network diagnostic tool on the ExaVault website.

The following example command and output shows ICMP-based MTR results:

mtr -n -c 100 192.168.52.10 --report
Start: Sat Oct 30 20:54:39 2021
HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.101.222               0.0%   100    0.7   0.7   0.6   0.9   0.0
  2.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.110.120.2               0.0%   100  266.5 267.4 266.4 321.0   4.8
  4.|-- 10.110.120.1              54.5%   100  357.6 383.0 353.4 423.7  19.6
  5.|-- 192.168.52.10             47.5%   100  359.4 381.3 352.4 427.9  20.6

The following example command and output shows TCP-based MTR results:

mtr -n -T -P 80 -c 100 192.168.52.10 --report
Start: Sat Oct 30 21:03:48 2021
HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.101.222               0.0%   100    0.9   0.7   0.7   1.1   0.0
  2.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.110.120.2               0.0%   100  264.1 265.8 263.9 295.3   3.4
  4.|-- 10.110.120.1               8.0%   100  374.3 905.3 354.4 7428. 1210.6
  5.|-- 192.168.52.10             12.0%   100  400.9 1139. 400.4 7624. 1384.3

The preceding examples show a Direct Connect connection with Border Gateway Protocol (BGP) peer 10.110.120.1 and 10.110.120.2. There's a loss percentage on the fourth and fifth destination hop that shows an issue with the Direct Connect connection or the remote router 10.110.120.1. Because a Direct Connect connection prioritizes TCP over ICMP, a TCP MTR result shows less loss percentage.

The following example command and output shows the local firewall or NAT device packet loss at 5%. The packet loss affects all subsequent hops, including the destination:

mtr -n -c 100 192.168.52.10 --report
Start: Sat Oct 30 21:11:22 2021
HOST:                              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.101.222               5.0%   100    0.8   0.7   0.7   1.1   0.0
  2.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.110.120.2               6.0%   100  265.7 267.1 265.6 307.8   5.1
  4.|-- 10.110.120.1               6.0%   100  265.1 265.2 265.0 265.4   0.0
  5.|-- 192.168.52.10              6.0%   100  266.7 266.6 266.5 267.2   0.0

Take a packet capture and analyze the results

Take a packet capture on the localhost and the EC2 instance. Use the tcpdump or Wireshark utility to get network traffic for analysis. For more information about tcpdump, see tcpdump on the tcpdump website. To download Wireshark, see the Wireshark download page on the Wireshark website.

For example, run the following tcpdump command to get the timestamp and host IP address:

tcpdump -i YOUR_NETWORK_INTERFACE  -s0 -w $(date +"%Y%m%d_%H%M%S").$(YOUR_HOSTNAME -s).pcap port YOUR_PORT

Note: Replace YOUR_NETWORK_INTERFACE with your network interface, YOUR_HOSTNAME with your host name, and YOUR_PORT with your port.

To calculate the network limit, bandwidth-delay product, and TCP buffer size, use the TCP throughput calculator on the Switch website. For more information, see Troubleshoot Direct Connect.

Related information

What's the difference between a hosted virtual interface (VIF) and a hosted connection?

Improving performance on AWS and hybrid networks