Amazon Linux Routing Challenge

Question

Hi Everyone, I have not worked with AWS for some time so pls bear with me.

I have the following topology:
* VPC 10.10.0.0/16
* Public subnet 10.10.0.0/24 containing an Amazon Linux instance (call it the 'Firewall') with a single NIC, IP addr = 10.10.0.150 and EIP = 3.45.120.240 (fictitious)
* Private subnet 10.10.1.0/24 with Windows RDS (10.10.1.10) and Windows AD DS (10.10.1.5)
* Route table applied to the public subnet routes 0.0.0.0/0 to IGW and 10.10.0.0/16 to local
* Route table applied to the private subnet routes 0.0.0.0/0 to the firewall NIC and 10.10.0.0/16 to local
* Kernel routing is enabled (ip_forward=1) and the filter table chains (input/forward/output) are configured to ACCEPT
* I use iptables on the firewall to DNAT inbound 3.45.120.240:3388 to 10.10.1.10:3389 and masquerade (SNAT) outbound
Basically, any instance in the private subnet routes in/out via the firewall with public IP address 3.45.120.240. I note private instances have a default gateway of 10.10.1.1. I assume this is an IP bound to the VPC?

I recently built a new firewall using Amazon Linux with the same config as the current (old) firewall, except private IP = 10.10.0.254 and EIP = 45.67.3.43 (fictitious). I want to replace this new with the old.

My issue: when I update the target for route 0.0.0.0/0 to the NIC of the new firewall, I do not see any traffic routing thru it (I am connected via SSH running tcpdump). If I start a repeating PING from Windows RDS to 1.1.1.1 and run tcpdump on the old firewall, I can see the ICMP flow, but when I flip the target for 0/0 route to the new firewall I see no packet flow. Both the old and new firewall have a default route 0.0.0.0/0 to 10.10.0.1. I assume this is the VPC router?

Packets from the private subnet do not appear to be reaching the new firewall and I do not know why. How do I troubleshoot further - is there a way to examine packet flow within the VPC? Thank you.

Accepted Answer

You don't mention this but just in case: Make sure that the source/destination check for your firewall instance is disabled as per our documentation for [NAT instances](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html).

Second: You can see where packets are being routed in a VPC by using [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html).

Third: VPC is an overlay network so doesn't necessarily behave quite the way that an on-premises network will. A good thing to watch here is [Another Day Another Billions Flows](https://youtu.be/8gc2DgBqo9U) but because that takes time - one of the things that happens in a VPC is that the flow path is cached so when you change routes you may not see the packets flow along that "new" route immediately. You might try stopping the existing `ping` or try sending ICMP echo requests to another destination.

Answer

For testing purposes I added a new route for my public home IP address with target of the new firewall and deployed a new Windows instance. It is working! Happy days.

Answer

Thank you for the response. The NAT instances link is helpful. On the old FW I have src/dst check stopped, however, on the new FW it is not stopped (like I said, it's been a while!). I will stop the src/dst check and try tomorrow morning AEST as I have a handful of users now logged on. Thanks again! :-)

Amazon Linux Routing Challenge

Relevant questions

VPC subnet routing.

VPC - Public/Private Subnets - Unable to access from internet

How do I set up Amazon VPC ingress routing with a stateless network appliance?

AWS Nitro Enclave instance does not have public ipv4 address

Subnets associated with a route table show as not associated with any RT

What is the VPC subnet route priority when two targets have the same default 0.0.0.0/0 route as the destination?

The route for the VPC not showing in iproute2 when a subnet has different mask

Associating DHCP options with a subnet instead of VPC?

Why VPC with a "public subnet only and AWS Site-to-Site VPN access" cannot be configured?