What tools can I use to troubleshoot network connectivity issues in Amazon VPC?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshoot Amazon VPC connectivity issues with Reachability Analyzer

Use Amazon VPC Reachability Analyzer to check for security group, network access control list (network ACL), and route table configurations that might cause connection issues. You can use Reachability Analyzer to analyze paths across multiple AWS accounts in your organization. For more information, see How Reachability Analyzer works.

Note: Amazon Q now enhances Amazon VPC network troubleshooting and allows users to diagnose connectivity issues with natural language queries.

Use VPC Flow Logs to identify connectivity issues

Activate Amazon VPC Flow Logs. Then, use Amazon VPC Flow Logs to monitor traffic to and from your VPC, subnet, or elastic network interface.

Analyze the data in your flow logs to understand traffic patterns and identify unexpected communication between resources. For more information, see Logging IP traffic using VPC Flow Logs.

If you store your flow logs in Amazon CloudWatch, then use Amazon CloudWatch Logs Insights to query your flow logs. You can also use CloudWatch Log Insights to analyze custom flow logs.

Run the following query to analyze your flow logs with CloudWatch Log Insights:

fields @timestamp, @message, interface-id, source-address, destination-address, action  
| filter @logStream like /^eni-/  
| filter action = "REJECT"  
| sort @timestamp desc

If you store your flow logs in Amazon Simple Storage Service (Amazon S3), then use Amazon Athena to analyze your flow logs. You can use Athena to run a predefined query for your flow logs. For more information, see Analyze VPC Flow Logs with point-and-click Athena integration.

Run the following query to analyze your flow logs with Athena:

SELECT  
interface_id,  
srcaddr,  
dstaddr,  
srcport,  
dstport,  
protocol,  
action  
FROM vpc_flow_logsWHERE action = 'REJECT'  
AND interface_id LIKE 'eni-%'  
AND date = 'yyyy-mm-dd'  
ORDER BY start DESC  
LIMIT 100;

Note: Replace yyyy-mm-dd with your date.

The previous queries list rejected traffic and the rejected traffic’s source and destination IP addresses, ports, and protocol. Use this information to identify and resolve misconfigured security group and network ACL rules.

Troubleshoot DNS Resolution problems

Use nslookup or dig to verify that DNS resolves correctly. Complete the following steps:

1. Run the following commands:

   nslookup example.com

   -or-

   dig example.com

   Note: Replace example.com with your IP address.
2. In the output, verify that the URL or hostname correctly resolves to the IP address.

Analyze Routing Tables

Troubleshoot issues with your Amazon VPC route tables. Verify that you configured the necessary routes for cross-network communications. Make sure that you correctly configured the default gateway configuration on your Amazon Elastic Compute Cloud (Amazon EC2) instance.

Run the following describe-route-tables command in the AWS CLI to check your route table’s configuration:

aws ec2 describe-route-tables --filters "Name=vpc-id,Values=example-vpc-id"

Note: Replace example-vpc-id with the ID of your VPC.

Run the following command to check the default gateway configuration on an instance:

route -n

Verify that the default gateway points to the correct address for your VPC.

Network Performance Tools

Use the following network performance tools to troubleshoot latency and packet loss issues within your VPC. If the following tools aren't installed, then use your system's package manager to install them.

Ping

Run the following ping command to test basic network connectivity

ping example.com

Note: Replace example.com with your URL.

In the following example output, there's network connectivity to example.com:

PING example.com (192.0.2.1): 56 data bytes  
64 bytes from 198.51.100.1: icmp_seq=0 ttl=242 time=297.779 ms  
64 bytes from 198.51.100.254: icmp_seq=1 ttl=242 time=296.183 ms

netcat

Run the following netcat command to test specific port connectivity from a machine that uses Linux operating systems (OS) to a destination:

nc -zv example.com example-port

Note: Replace example.com with your URL. Replace example-port with your port.

In the following example output, connection to example.com is successful on port 443 (HTTPS):

Ncat: Version 7.93 ( https://nmap.org/ncat )  
Ncat: Connected to 192.0.2.0:443.  
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

In the following example output, connection to example.com is unsuccessful on port 8080:

Ncat: Version 7.93 ( https://nmap.org/ncat )  
Ncat: Connection to 198.51.100.1 failed: TIMEOUT.  
Ncat: Trying next address...  
Ncat: Connection to  198.51.100.254 failed: TIMEOUT.

Run the following netcat command to use the time parameter to measure TCP connection timing:

time nc -zv example.com example-port

Note: Replace example.com with your URL. Replace example-port with your port.

curl

Run the following curl command to measure connection timing details for connection stages, including DNS lookup, TCP handshake, and data transfer:

curl -kso /dev/null https://example.com -w "==============\n\n | dnslookup: %{time_namelookup}\n | connect: %{time_connect}\n | TLS-Handshake: %{time_appconnect}\n | pretransfer: %{time_pretransfer}\n | starttransfer: %{time_starttransfer}\n | total: %{time_total}\n | size: %{size_download}\n | HTTPCode=%{http_code}\n\n"

Note: Replace example.com with your URL.

traceroute

Run one of the following commands to analyze network paths:

For Linux:

traceroute example-ip

For Windows:

tracert example-ip

Note: Replace example-ip with the destination IP address or hostname.

My Traceroute

Run the following mtr command to run a real-time path analysis:

mtr example-ip

Note: Replace example-ip with the destination IP address or hostname.

For more information, see Diagnosing packet loss and latency with mtr.

iPerf

Use iPerf to measure network performance metrics between two instances. For more information, see iPerf on the official iPerf website. Complete the following steps:

1. Configure your security groups to allow traffic on port 5201 between the instances. Port 5201 is the default iperf port.

2. On the server instance, start iperf in server mode:

   iperf3 -s

3. On the client instance, run the test:

   iperf3 -c example_ip

   Note: Replace example-ip with the server instance’s private IP address.

For more information, see How do I benchmark network throughput between Amazon EC2 Linux instances in the same VPC?

tcpdump

Run the following tcpdump command to capture network packets on your Amazon EC2 Linux instance. Make sure you have enough disk space to store the packet capture file:

tcpdump -v host example-ip -c -w output.pcap

Note: Replace example-ip with the destination IP address or hostname.

In the output, check the following to identify performance issues:

• To identify specific TCP streams experiencing high latency, review the time between SYN and SYN-ACK packets.
• To detect intermittent latency spikes, correlate packet timestamps with the performance issues that you find.
• Verify that the destination correctly responds, even when flow logs show an ACCEPT status.

Internet performance monitoring with Internet Monitor

Create and configure an Amazon CloudWatch Internet Monitor, and then use it to monitor your internet performance. For more information, see Introducing Amazon CloudWatch Internet Monitor.

Note: Make sure that you use a supported AWS Region before you set up Internet Monitor. Internet Monitor is available only in select Regions.

Monitor ENA metrics

Monitor your instance’s Elastic Network Adapter (ENA) metrics. Verify that your instance’s allowance_exceeded ENA driver metric values are 0.

Perform additional troubleshooting steps

If the previous methods don't resolve your issue, then take the following actions:

• If you experience issues with specific network traffic flows, especially containerized workloads or applications running on AWS Fargate, then activate VPC Traffic Mirroring to capture and analyze the traffic.
• Verify that your security group rules, network ACLs, and route tables allow all necessary traffic.
• Analyze the Amazon CloudWatch metrics for your Amazon EC2 instance, including CPU usage, memory usage, and disk I/O, and verify that you have sufficient resources.
• Verify that your Amazon EC2 instance uses the appropriate instance type and network performance configuration for your workload requirements.