Custom NAT instance does not forward requests cross AZ

Question

Hello,

I have a custom NAT instance that is supposed to route internet for instances in two subnets.

The instance is set up simply using user data + eth1 with source/destination check off.

```bash
  #!/bin/bash
  sysctl -w net.ipv4.ip_forward=1
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
  iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT
```

* NAT is in AZ-B.
* Private instance in AZ-B can connect to the internet.
* Private Instance in AZ-C cannot connect.
* All instances can SSH to each other.

By further investigation it seems that the problem lays somewhere inside the guest OS. The packets from subnet C are visible on eth1 with tcpdump and Wireshark.

Route table is the same for both private subnets. This is the flow log I collected by curling to google.com. I simplified it, so that B and C = private IPs in respective AZ. Nothing seems to be rejected on any end.

| @time              | srcAddr    | srcPort | dstAddr    | dstPort | action | interfaceId |
|-------------------------|------------|---------|------------|---------|--------|-------------|
| 09:46:11 | google.com | 80      | Public NAT | 60410   | ACCEPT | Public NAT  |
| 09:46:11 | Public NAT | 60410   | google.com | 80      | ACCEPT | Public NAT  |
| 09:45:41 | B          | 60410   | google.com | 80      | ACCEPT | Private B   |
| 09:45:41 | google.com | 80      | B          | 60410   | ACCEPT | Private B   |
| 09:45:39 | NAT        | 80      | B          | 60410   | ACCEPT | Private NAT |
| 09:45:39 | B          | 60410   | NAT        | 80      | ACCEPT | Private NAT |
|  09:41:25 | C          | 40872   | google.com | 80      | ACCEPT | Private C   |
|  09:41:23 | C          | 40872   | NAT        | 80      | ACCEPT | Private NAT |
|  09:40:09 | C          | 48212   | NAT        | 80      | ACCEPT | Private NAT |
| 09:39:52. | C          | 48212   | google.com | 80      | ACCEPT | Private C   |

Accepted Answer

Based off your routes provided it makes logical sence to add a new route to the NAT instance as follows for the other subnet

I am assuming eni-04682433028b75d56 is eth1 on the NAT device

```
10.63.3.0/24 via 10.63.2.1 dev eth1
```

Answer

Please can you provide route tables for the subnets in question.

Can you provide route table inside of the NAT instance also.

Your NAT instance will need a route back to subnet C out of the correct interface too. I’ve a feeling your default route inside your NAT instance my be breaking the return traffic to subnet C. Without seeing all the routes it’s hard to be sure. 
I assume the instances on subnet b are on the same subnet as the NAT instance. If so then naturally the route table routes subnet b out of the correct interface where as subnet c is taking the default route.

Thanks

Answer

Surely, thanks for the suggestion!

### Main rtb:

| Destination | Target |
| --- | --- |
| 2a05:d014:97d:e000::/56	 | local |
| 10.63.0.0/16 | local |

### For public subnet:

| Destination | Target |
| --- | --- |
| ::/0  | igw-07ee2278a218a671f |
| 2a05:d014:97d:e000::/56 | local |
| 0.0.0.0/0 | igw-07ee2278a218a671f |
| 10.63.0.0/16 | local |

### For private subnets (the same for both):

| Destination | Target |
| --- | --- |
| ::/0  | eigw-0f26feb7612f0017b |
| 2a05:d014:97d:e000::/56 | local |
| 0.0.0.0/0 |  eni-04682433028b75d56 |
| 10.63.0.0/16 | local |

Routes on Linux - private B
```
default via 10.63.2.1 dev eth0 
10.63.2.0/24 dev eth0 proto kernel scope link src 10.63.2.164 
169.254.169.254 dev eth0
```

Routes on Linux - private C
```
default via 10.63.3.1 dev eth0 
10.63.3.0/24 dev eth0 proto kernel scope link src 10.63.3.135 
169.254.169.254 dev eth0
```

Routes on Linux - NAT
```
default via 10.63.1.1 dev eth0 
default via 10.63.2.1 dev eth1 metric 10001 
10.63.1.0/24 dev eth0 proto kernel scope link src 10.63.1.244 
10.63.2.0/24 dev eth1 proto kernel scope link src 10.63.2.159 
169.254.169.254 dev eth0
```

@time	srcAddr	srcPort	dstAddr	dstPort	action	interfaceId
09:46:11	google.com	80	Public NAT	60410	ACCEPT	Public NAT
09:46:11	Public NAT	60410	google.com	80	ACCEPT	Public NAT
09:45:41	B	60410	google.com	80	ACCEPT	Private B
09:45:41	google.com	80	B	60410	ACCEPT	Private B
09:45:39	NAT	80	B	60410	ACCEPT	Private NAT
09:45:39	B	60410	NAT	80	ACCEPT	Private NAT
09:41:25	C	40872	google.com	80	ACCEPT	Private C
09:41:23	C	40872	NAT	80	ACCEPT	Private NAT
09:40:09	C	48212	NAT	80	ACCEPT	Private NAT
09:39:52.	C	48212	google.com	80	ACCEPT	Private C

Destination	Target
::/0	igw-07ee2278a218a671f
2a05:d014:97d:e000::/56	local
0.0.0.0/0	igw-07ee2278a218a671f
10.63.0.0/16	local

Destination	Target
::/0	eigw-0f26feb7612f0017b
2a05:d014:97d:e000::/56	local
0.0.0.0/0	eni-04682433028b75d56
10.63.0.0/16	local

Custom NAT instance does not forward requests cross AZ

Main rtb:

For public subnet:

For private subnets (the same for both):

Relevant content