VPC Peering Connection across different region, ping not responding

0

I have an EC2 Instance in ap-south-1 and in eu-west-2 and I am trying to setup a Peering VPC connection to ping from one instance to the other.

Overview

Routing Table for eu-west-2 VPC

DestinationTarget
::/0igw-aaa
0.0.0.0/0igw-aaa
2a05:d01c:273:3700::/56local
172.31.0.0/16local
10.0.16.0/20pcx-ccc

Routing Table for ap-south-1 VPC

DestinationTarget
::/0igw-bbb
0.0.0.0/0igw-bbb
2406:da1a:1d5:f000::/56local
10.0.0.0/16local
172.31.32.0/20pcx-ccc

Security Group in eu-west-2

IP versionTypeProtocolPort rangeSource
IPv4All trafficAllAll10.0.0.0/16

Security Group in ap-south-1

IP versionTypeProtocolPort rangeSource
IPv4All trafficAllAll172.31.32.0/20

I have verified the peering connection is Activated between the two VPCs/subnets. Both regions Network ACLs allow all traffic in and out, similarly the security groups allow all outbound traffic. However, I try and do a simple "ping" to the other instance, no packets are received.

Is there anything wrong with the setup? There is one thing I am unsure about... Is it possible to have an Internet Gateway and a Peering Connection on the same subnet?

Looking at the routing tables, the IP 172.31.40.90 will satisify the 172.31.32.0/20 CIDR block (for the peering connection), and the 0.0.0.0/0 CIDR block for the Internet Gateway... how does it know which route to take if the IP satisfies more than 1 route in the Routing Table?

Update The iptables config on the eu-west-2 was the offending issue.

I reset the iptables using the following commands

sudo iptables -P INPUT ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -F

The offending iptables -L config

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_ACCEPT  all  --  anywhere             anywhere
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_ACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:914
AS0_ACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:915
AS0_ACCEPT  udp  --  anywhere             anywhere             state NEW udp dpt:916
AS0_ACCEPT  udp  --  anywhere             anywhere             state NEW udp dpt:917
AS0_WEBACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_WEBACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:943
AS0_APIACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_APIACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:945

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_OUT_S2C  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
AS0_OUT_LOCAL  all  --  anywhere             anywhere

Chain AS0_ACCEPT (7 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain AS0_APIACCEPT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain AS0_DNS (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             ip-172-31-0-2.eu-west-2.compute.internal
RETURN     all  --  anywhere             anywhere

Chain AS0_IN (4 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             ip-10-0-0-1.eu-west-2.compute.internal
AS0_IN_POST  all  --  anywhere             anywhere

Chain AS0_IN_NAT (0 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere             MARK or 0x8000000
ACCEPT     all  --  anywhere             anywhere

Chain AS0_IN_POST (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             ip-172-31-32-0.eu-west-2.compute.internal/20
AS0_OUT    all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain AS0_IN_PRE (2 references)
target     prot opt source               destination
AS0_DNS    tcp  --  anywhere             anywhere             state NEW tcp dpt:domain
AS0_DNS    udp  --  anywhere             anywhere             state NEW udp dpt:domain
AS0_IN     all  --  anywhere             169.254.0.0/16
AS0_IN     all  --  anywhere             ip-192-168-0-0.eu-west-2.compute.internal/16
AS0_IN     all  --  anywhere             ip-172-16-0-0.eu-west-2.compute.internal/12
AS0_IN     all  --  anywhere             ip-10-0-0-0.eu-west-2.compute.internal/8
ACCEPT     all  --  anywhere             anywhere

Chain AS0_IN_ROUTE (0 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere             MARK or 0x4000000
ACCEPT     all  --  anywhere             anywhere

Chain AS0_OUT (2 references)
target     prot opt source               destination
AS0_OUT_POST  all  --  anywhere             anywhere

Chain AS0_OUT_LOCAL (1 references)
target     prot opt source               destination
DROP       icmp --  anywhere             anywhere             icmp redirect
ACCEPT     all  --  anywhere             anywhere

Chain AS0_OUT_POST (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere

Chain AS0_OUT_S2C (1 references)
target     prot opt source               destination
AS0_OUT    all  --  anywhere             anywhere

Chain AS0_WEBACCEPT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
asked 7 months ago619 views
3 Answers
1
Accepted Answer

Ping needs an ICMP rule in the security groups, not an IPv4 rule.

The eu-west-2 security group needs an outbound rule to target 10.0.0.0/16 for ICMP, and also an inbound rule from source 10.0.0.0/16 for ICMP.

And similar for ap-south-1, but using 172.31.32.0/20.

profile picture
EXPERT
Steve_M
answered 7 months ago
  • I've added an "All ICMP - IPv4" rule to Inbound and Outbound SGs in both regions with the correct CIDRs, unfortunately still not working :(

  • Anything on the EC2 instances that could be blocking ping? Such as a host-based firewall like iptables, firewalld or ufw on Linux, or Windows Firewall?

  • +1 to this last comment - check the host firewall.

  • Intrestingly, I can ping the public IP address, but not the private IP address. Why would that be?

  • It was the iptables on the eu-west-2 instance... I shall edit my post and post the offending iptables config... thanks everyone

0

The router will use the more specific rule.

Can you try using the VPC Reachability Analyzer to see if it sheds some light?

profile pictureAWS
EXPERT
kentrad
answered 7 months ago
  • Yes I have used the VPC Reachability Analyzer.. here is the result

    NO_POSSIBLE_DESTINATION: The network component vpc-aaa cannot deliver the packet to any possible destination, or the network component is sending traffic towards a different account or region. See documentation to enable Reachability Analyzer for cross-account analyses.

    Interestingly, if I test it one way it succeeds, if I test it another way it fails... But neither pings work from either side The network is sending traffic to a different region, so I thought this is a limitation of the Reachability Analyzer not being able to analyze across regions.

  • Sorry, missed that limitation.

  • Can you trace from the instance to the peering connection from both sides?

  • Also, I would try opening up the SGs on both side all the way, both incoming and outgoing, to eliminate them as the cause.

  • So, I have opened up the SGs temporarily on both sides, both In and Out-going. So I don't think the SGs or the AGLs are the cause... I think the problem is with the routing table. Could it be because I am confusing Public and Private routing tables and subnets? Do I need a subnet to route to the Internet Gateway, and a separate subnet to route to the Peer Connection or can I use a routing table for both private and public?

0

Hello,

Let me help you with this scenario.

Can you provide me below information:

  • Source IP:
  • Destination IP:
  • Source side of Outbound security group rules, Destination side of Inbound security group rules.
  • Inbound and Outbound NACLs rules.
  • Does both the route table have route pointing out to destination VPC via PCX (VPC peering)?
  • How you are trying to access? Is it IPv4 ICMP request?
  • Have you tried Telnet?
  • Is destination instance windows? Can you verify the OS level firewall rules?
  • Can you launch Linux instance in destination subnet and then try to reach that. What do you see?
  • Can you run a TCPdump on linux box? Do you see incoming packets? #tcpdump -i eth0 src x.x.x.x
  • If you still observing the issue in that case enable flow logs and check for rejected packet.

Thank you.

profile picture
EXPERT
answered 7 months ago
  • Source IP: 172.31.40.90 (private in eu-west-2) Destination IP: 10.0.18.69 (private in ap-south-1) Source Outbound SG: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Destination Inbound SG: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Source ACL: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Destination ACL: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Source Route Table: 10.0.16.0/20 Points to PCX Destination Route Table: 172.31.32.0/20 Points to PCX Yes, ICMP (ping req). No telnet. Both instances Linux. I'll verify firewall

  • Above configuration looks good, can you try these steps as well:

    • Verify route is pointed towards correct peering.
    • Can you run a TCPdump on linux box? Do you see incoming packets? #tcpdump -i eth0 src x.x.x.x. (Try bidirectional)
    • If you still observing the issue in that case enable flow logs and check for rejected packet.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions