How do I troubleshoot connectivity issues from Direct Connect to AWS resources?

Resolution

Check the status of your virtual interface

If your virtual interface stops working, then take the following actions:

• Check the AWS Health Dashboard for ongoing or recently completed AWS maintenance that might affect your Direct Connect connection or virtual interface.
• Check the status of your virtual interface. If the status is DOWN, then the Border Gateway Protocol (BGP) peering session didn't establish. To troubleshoot BGP issues, see Troubleshoot Direct Connect.

Troubleshoot issues for private virtual interfaces

Configure inbound and outbound rules

Configure inbound and outbound rules for your destination instance's security group and subnet's network access control list (network ACL). The rules must allow bidirectional connectivity between AWS and on-premises resources.

Note: The specific rules depend on your source IP address, destination IP address, and port. If you have an on-premises inspection firewall, then configure it to allow bidirectional traffic.

Check your BGP routing configuration

For the BGP routing configuration of your on-premises router, make sure that you direct the required routes toward AWS.

If you activated route propagation on your Amazon Virtual Private Cloud (Amazon VPC) route table, then make the routes visible on the route table. In your Amazon VPC route tables, your on-premises routes must point toward the virtual gateway as the next hop or destination. For example, if the on-premises router advertises 10.0.0.0/8 and points to AWS, then your route table includes a route entry for 10.0.0.0/8. The entry points toward the virtual gateway that's associated with your VPC as the next hop or destination.

Check your on-premises router

Make sure that your on-premises router receives routes for the VPC CIDR over the BGP. The router must receive routes from the IP address of the AWS peer that's associated with the Direct Connect virtual interface.

If you don't receive routes from the AWS peer's IP address, then associate your virtual private gateway to the correct VPC.

If your private virtual interface ends on the Direct Connect gateway, then associate the correct virtual private gateway with your Direct Connect gateway. Configure the allowed prefixes so that the VPC CIDR directs toward the on-premises router.

Run a traceroute

To run a traceroute based on bidirectional Internet Control Message Protocol (ICMP) from your on-premises router to your VPC instance, run the following command:

sudo traceroute -n -I YOUR_IP_ADDRESS

Note: Replace YOUR_IP_ADDRESS with the private IP address of your Amazon Elastic Compute Cloud (Amazon EC2) instance or on-premises host.

If your on-premises router or firewall blocks ICMP-based traceroute requests, then run the following TCP-based traceroute on the appropriate TCP port:

sudo traceroute -n -T -p 22 YOUR_IP_ADDRESS

Note: Replace YOUR_IP_ADDRESS with your private IP address. In the preceding command, -n -T -p 22 runs a traceroute on port 22. You can use any port that your application uses as a listener.

Check your traceroute results

Check the visibility and behavior of the on-premises router and AWS peer IP addresses that are associated with your virtual interface.

Based on your scenario, take the following actions:

• The traceroute goes from AWS to on-premises resources and the traffic stops at the on-premises router's IP address: Configure your on-premises network firewall settings to allow bidirectional connectivity on your port.
• The traceroute goes from on-premises resources to AWS and the traffic stops at the AWS peer's IP address: Check your network ACL and security configurations. You can also use VPC Flow Logs to check whether a specific elastic network interface received the packets that the on-premises router sent.
• The AWS or on-premises peer IP address isn't associated with the virtual interface and traffic forwards over an incorrect path: Confirm that the on-premises router has a more specific or preferred route for the same CIDR through a different peer.
• The traceroute from AWS to the on-premises router doesn't contain the AWS peer's IP address: Look for a secondary path, such as a virtual interface or VPN, with traffic that flows toward your on-premises router. For example, another virtual interface might end on the same virtual private gateway, or the Direct Connect gateway advertises the same on-premises route. Or, existing AWS Site-to-Site VPN connections might advertise specific routes for your on-premises router on the VPC route table.

Compare traceroutes from AWS to the on-premises router and from the on-premises router to AWS. If both traceroutes have different hops, then the routing is asymmetric. Use routing policies to make sure that the same Direct Connect private virtual interface is preferred bidirectionally.

Troubleshoot issues for public virtual interfaces

Validate routes from public prefixes

Verify that the on-premises router that's hosting your public virtual interface receives routes from public prefixes from the AWS peer's IP address. If you have an inbound prefix filter and route map to filter the routes, then make sure that the prefix filter matches the required prefixes.

Advertise the public peer IP address

If you perform NAT for your on-premises networks, then advertise your public peer IP address to AWS over the BGP.

Note: Make sure that you connect from a prefix that you advertised from on-premises resources to AWS over the public virtual interface. You can't connect from a prefix that you didn't advertise to a public virtual interface. Also, you must add prefixes to the "Prefixes you want to advertise" list before you can advertise them over BGP on the public virtual interface. The prefix in the list must be the same or less specific than what you plan to advertise.

Run a traceroute

Run a traceroute from your on-premises resources to AWS, and then verify that your traffic forwarded over the Direct Connect public virtual interface. If traffic forwards over the public virtual interface, then the traceroute includes the IP addresses that are associated with the virtual interface.

If you must check the network path that AWS uses, then launch a public Amazon EC2 instance. The instance must have the same AWS Region as your AWS service. After you launch the instance, run a traceroute to the on-premises router. If the traceroute shows that traffic forwards over the internet or through a different virtual interface, then a specific route might be advertised.

Note: AWS uses AS_PATH and Longest Prefix Match to determine the routing path. Direct Connect is the preferred path for traffic sourced from AWS. For more information, see Public virtual interface routing policies.

Test the connectivity to a public AWS service

Verify that your connectivity to a public AWS service, such as Amazon Simple Storage Service (Amazon S3), is working for the correct destination Region. Then, confirm that you have a BGP community tag on the public prefixes that you advertise to AWS. For more information, see Public virtual interface BGP communities.

Note: BGP community tags determine how far to propagate your prefixes on the AWS network.

Troubleshoot issues for transit virtual interfaces

Configure an on-premises CIDR to a transit gateway route

If you didn't configure a route from your on-premises CIDR to a transit gateway for your destination resource's VPC subnet route table, then configure one. Also, configure the instance or resource security groups and the subnet's network ACL to allow bidirectional connectivity. For more information, see Network ACLs for transit gateways in AWS Transit Gateway.

Validate BGP routes

Verify that the on-premises router that's associated with your transit virtual interface receives the correct routes over the BGP from the AWS peer. The routes are for the destination VPC CIDR.

If you don't receive the required routes, then check the allowed prefixes list. Configure the allowed prefixes for the Direct Connect gateway that's associated with transit gateway with the required prefixes. AWS advertises only routes that you add to the allowed prefixes list over a transit virtual interface.

Validate on-premises network prefixes

For the on-premises router that's associated with the transit virtual interface, advertise the required on-premises network prefixes to AWS. If you propagate routes from the Direct Connect gateway to a transit gateway route table, then make the routes visible on the transit gateway route table. If the routes aren't visible, then check the AS_PATH on the advertised routes. The AS_PATH must not include the transit gateway ASN.

Note: You can't propagate routes from a transit gateway to a VPC. So, you must verify that your VPC route table has route entries that point toward the transit gateway. If you advertise the route that contains the transit gateway ASN on the AS_PATH, then the route doesn't install on the route table. Make sure that your on-premises router and transit gateway use a different ASN.

Validate transit gateway routes

Check that the transit gateway table that's associated with the Direct Connect gateway and destination VPC attachments have the correct route for the destination. The transit gateway route table that's associated with the Direct Connect gateway must have a route for the VPC CIDR that you direct to the VPC attachment. The transit gateway route table that's associated with the VPC attachment must have a route for the on-premises CIDR that you direct to the Direct Connect gateway attachment.

Validate Direct Connect gateway settings

If the traceroute from AWS to on-premises resources doesn't include your virtual interface's peer IP address, then check your Direct Connect gateway settings. You must have other transit virtual interfaces on the same Direct Connect gateway that advertise the same on-premises routes. To identify the virtual interface that you must use for outbound connectivity, check the transit virtual interface routing policy.

Check subnet associations for VPC attachments

Confirm that you associated a subnet with the transit gateway VPC attachment in the same Availability Zone as the destination resource. For example, your transit gateway VPC attachment must have one subnet in the same Availability Zone as your instance.

Note: To identify bidirectional traffic on the network interface, check VPC flow logs to verify that your on-premises traffic reaches a specific instance's network interface.