AWS EC2 secondary ip outbond traffic

Question

Hello,
I added a secondary private ipv4 to an EC2 instance/ENI (for testing purposes used also default Amazon Linux 2 AMI which should handle such setups) but while it is reachable from within the VPC/other instances no outbond connections outside VPC can be made:

```
[root@ip-172-31-28-154 ~]# curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/02:42:36:f4:99:c7/local-ipv4s

172.31.28.154
172.31.17.114
```

```
[root@ip-172-31-28-154 ~]# ip a

2: enX0:  mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:42:36:f4:99:c7 brd ff:ff:ff:ff:ff:ff
    altname eni-01bd21ce251e90926
    altname device-number-0
    inet 172.31.17.114/32 scope global noprefixroute enX0
       valid_lft forever preferred_lft forever
    inet 172.31.28.154/20 metric 512 brd 172.31.31.255 scope global dynamic enX0
       valid_lft 3252sec preferred_lft 3252sec

[root@ip-172-31-28-154 ~]# ip route
default via 172.31.16.1 dev enX0 proto dhcp src 172.31.28.154 metric 512
172.31.0.2 via 172.31.16.1 dev enX0 proto dhcp src 172.31.28.154 metric 512
172.31.16.0/20 dev enX0 proto kernel scope link src 172.31.28.154 metric 512
172.31.16.1 dev enX0 proto dhcp scope link src 172.31.28.154 metric 512

```

While the primary works 
```
[root@ip-172-31-28-154 ~]# curl -I  --interface 172.31.28.154 google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
...
```
.. the secondary doesn't 
```
 [root@ip-172-31-28-154 ~]# curl -I --connect-timeout 11 --interface 172.31.17.114 google.com
curl: (28) Failed to connect to google.com port 80 after 11002 ms: Timeout was reached
```

Is it supposed to be like that?

I have read several guides:
https://repost.aws/knowledge-center/secondary-private-ip-address
https://repost.aws/knowledge-center/ec2-ubuntu-secondary-network-interface

.. but even specifically setting up multiple route tables 
```
ip route add default via 172.31.16.1 dev enX0 table 1000
ip route add 172.31.17.114 dev enX0 table 1000
ip rule add from 172.31.17.114 lookup 1000
```
doesn't help.

Gary Mclean · Answer

You can only have one default route via 1 interface. You have to choose which interface it is. If you have separate interfaces on different subnets then you can have different route for different subnets.

You have to be careful of asynchronous routing however.

Leo K · Answer

It could be a routing issue as Oleksii Bebych explained, but in addition to that, note that security groups are attached separately to each ENI. You might want to check if you have the same security groups attached to the secondary ENI as to the primary one. You can modify the security group attachments for each ENI separately in the console in the "Network interfaces" view. This design can be misleading, because there's a security group selection also available for an EC2 instance, despite it not guaranteeing that the same groups are used for all the attached ENIs.

Another simple thing is that if your routing is configured or working such that traffic from either IP address might exit through the same interface, your options would be either to configure the operating system to send traffic from each source IP via the correct ENI, or alternatively, disable the "source/destination check" option for the ENI through which outbound traffic is exiting. That would allow traffic with an IP address not matching that specific ENI's addresses to exit.

In terms of capacity, which ENI is used to send which packet doesn't matter. Bandwidth and other limits are applied by EC2 instance, regardless of how many physical-looking ENIs it has. They're all fully virtual.

Oleksii Bebych · Answer

**Add a new routing table entry for the secondary IP**:
```
sudo ip route add default via 172.31.16.1 dev enX0 table 1000
sudo ip route add 172.31.16.0/20 dev enX0 proto kernel scope link src 172.31.28.154 table 1000
sudo ip route add 172.31.17.114 dev enX0 table 1000
```

**Add a rule to use the new routing table for the secondary IP**:
```
sudo ip rule add from 172.31.17.114 lookup 1000
```

**Persist the Configuration**

To ensure that these configurations persist across reboots, you need to add these commands to your network configuration scripts.

For Amazon Linux 2, you can place these commands in a script and configure it to run at startup.

Example Script /etc/sysconfig/network-scripts/route-enX0

Create a custom script for the secondary IP route:
```
sudo vi /etc/sysconfig/network-scripts/route-enX0
```
Add the following content:
```
172.31.16.0/20 dev enX0 table 1000
172.31.16.1 dev enX0 table 1000
default via 172.31.16.1 dev enX0 table 1000
```

Example Script /etc/sysconfig/network-scripts/rule-enX0

Create a custom script for the routing rules:

```
sudo vi /etc/sysconfig/network-scripts/rule-enX0
```
Add the following content:

```
from 172.31.17.114/32 lookup 1000
```

**Restart Network Service**

After creating these files, restart the network service to apply the changes:
```
sudo systemctl restart network
```

**Verify Configuration**

Verify that the rules and routes are correctly applied:
```
ip rule show
ip route show table 1000
```

**Test Connectivity**

Test outbound connectivity from the secondary IP:
```
curl -I --interface 172.31.17.114 google.com
```

Leo K · Answer

Could you perhaps check with `tcpdump` what's happening at the network interface level when trying to connect unsuccessfully to the internet via the secondary IP. Something like: `tcpdump -nnn -p -i enX0 -c 10000 not port 22`

Also, did the ENI have an Elastic IP or automatically allocated public IP assigned to it, or is ÌPv4 traffic to the internet routed via a NAT gateway?

Roze · Answer

> Add a new routing table entry for the secondary IP:
> Add a rule to use the new routing table for the secondary IP:

For some reason doesn't work:

```
[root@ip-172-31-28-154 ~]# ip rule list
0:      from all lookup local
32765:  from 172.31.17.114 lookup 1000
32766:  from all lookup main
32767:  from all lookup default

[root@ip-172-31-28-154 ~]# ip route show table 1000
default via 172.31.16.1 dev enX0
172.31.16.0/20 dev enX0 proto kernel scope link src 172.31.28.154
172.31.17.114 dev enX0 scope link

[root@ip-172-31-28-154 ~]# curl -I --connect-timeout 5 --interface 172.31.17.114 google.com
curl: (28) Failed to connect to google.com port 80 after 5002 ms: Timeout was reached
```

The route table is present but still not possible to use the secondary ip for anything outside, though as I wrote it works just fine within the VPC:

```
[root@ip-172-31-28-154 ~]# ssh -b 172.31.17.114  172.31.29.82
Last login: Wed Jun 19 23:18:05 2024 from 172.31.17.114
Have a lot of fun...

www1:~ # who
root     pts/0        Jun 19 23:18 (172.31.17.114)
```

> It could be a routing issue as Oleksii Bebych explained, but in addition to that, note that security groups are attached separately to each ENI.

It's a secondary ip for the same ENI as in the instance has only one network interface so the same security group/rules should apply (also for testing purposes outgoing it's also fully open). 
Have also disabled source/destination checks.

Roze · Answer

> what's happening at the network interface level when trying to connect unsuccessfully to the internet via the secondary IP

For:
curl -I --connect-timeout 5 --interface 172.31.17.114 google.com
```
21:42:47.206277 IP 172.31.28.154.48637 > 172.31.0.2.53: 55224+ A? google.com. (28)
21:42:47.206297 IP 172.31.28.154.48637 > 172.31.0.2.53: 29117+ AAAA? google.com. (28)
21:42:47.206836 IP 172.31.0.2.53 > 172.31.28.154.48637: 29117 1/0/0 AAAA 2a00:1450:4001:828::200e (56)
21:42:47.206836 IP 172.31.0.2.53 > 172.31.28.154.48637: 55224 1/0/0 A 142.250.184.238 (44)
21:42:47.207162 IP 172.31.17.114.40278 > 142.250.184.238.80: Flags [S], seq 137003124, win 62727, options [mss 8961,sackOK,TS val 3586508624 ecr 0,nop,wscale 7], length 0
21:42:48.222004 IP 172.31.17.114.40278 > 142.250.184.238.80: Flags [S], seq 137003124, win 62727, options [mss 8961,sackOK,TS val 3586509639 ecr 0,nop,wscale 7], length 0
21:42:50.301996 IP 172.31.17.114.40278 > 142.250.184.238.80: Flags [S], seq 137003124, win 62727, options [mss 8961,sackOK,TS val 3586511719 ecr 0,nop,wscale 7], length 0
21:42:50.405966 IP 172.31.28.154.44534 > 169.254.169.123.123: NTPv4, Client, length 48
21:42:50.406253 IP 169.254.169.123.123 > 172.31.28.154.44534: NTPv4, Server, length 48

```

For the working primary ip:

[root@ip-172-31-28-154 ~]# curl -I --connect-timeout 5 --interface 172.31.28.154 google.com

```
21:45:15.599125 IP 172.31.28.154.58153 > 172.31.0.2.53: 43707+ A? google.com. (28)
21:45:15.599139 IP 172.31.28.154.58153 > 172.31.0.2.53: 27582+ AAAA? google.com. (28)
21:45:15.599710 IP 172.31.0.2.53 > 172.31.28.154.58153: 43707 1/0/0 A 142.250.184.238 (44)
21:45:15.599710 IP 172.31.0.2.53 > 172.31.28.154.58153: 27582 1/0/0 AAAA 2a00:1450:4001:828::200e (56)
21:45:15.600022 IP 172.31.28.154.55032 > 142.250.184.238.80: Flags [S], seq 2667010660, win 62727, options [mss 8961,sackOK,TS val 3842286745 ecr 0,nop,wscale 7], length 0
21:45:15.601159 IP 142.250.184.238.80 > 172.31.28.154.55032: Flags [S.], seq 790944820, ack 2667010661, win 65535, options [mss 1412,sackOK,TS val 2618665625 ecr 3842286745,nop,wscale 8], length 0
21:45:15.601176 IP 172.31.28.154.55032 > 142.250.184.238.80: Flags [.], ack 1, win 491, options [nop,nop,TS val 3842286746 ecr 2618665625], length 0
21:45:15.601225 IP 172.31.28.154.55032 > 142.250.184.238.80: Flags [P.], seq 1:75, ack 1, win 491, options [nop,nop,TS val 3842286746 ecr 2618665625], length 74: HTTP: HEAD / HTTP/1.1
21:45:15.602312 IP 142.250.184.238.80 > 172.31.28.154.55032: Flags [.], ack 75, win 256, options [nop,nop,TS val 2618665626 ecr 3842286746], length 0
21:45:15.616094 IP 142.250.184.238.80 > 172.31.28.154.55032: Flags [P.], seq 1:555, ack 75, win 256, options [nop,nop,TS val 2618665640 ecr 3842286746], length 554: HTTP: HTTP/1.1 301 Moved Permanently
21:45:15.616103 IP 172.31.28.154.55032 > 142.250.184.238.80: Flags [.], ack 555, win 487, options [nop,nop,TS val 3842286761 ecr 2618665640], length 0
21:45:15.616759 IP 172.31.28.154.55032 > 142.250.184.238.80: Flags [F.], seq 75, ack 555, win 487, options [nop,nop,TS val 3842286761 ecr 2618665640], length 0
21:45:15.618355 IP 142.250.184.238.80 > 172.31.28.154.55032: Flags [F.], seq 555, ack 76, win 256, options [nop,nop,TS val 2618665642 ecr 3842286761], length 0
21:45:15.618366 IP 172.31.28.154.55032 > 142.250.184.238.80: Flags [.], ack 556, win 487, options [nop,nop,TS val 3842286763 ecr 2618665642], length 0
```

> Also, did the ENI have an Elastic IP or automatically allocated public IP assigned to it, or is ÌPv4 traffic to the internet routed via a NAT gateway?

Initially I tested on instance which has assigned Elastic IP (running OpenSuse), but since I couldn't get it to work there I made an Amazon Linux 2 instance which has automatically assigned primary (+public ip) and secondary private ipv4 ips.

Leo K · Answer

The way public IPs assigned to ENIs work in general is that packets arriving to the public IP have their destination translated to the primary private IP of the ENI. Packets leaving via an IGW get their source IP translated to the public IP, whether it's automatically assigned or a static elastic IP.

I suspect the translation also works in the outbound direction only for the primary private IP of the ENI, albeit I can't find an explicit statement confirming or denying this from documentation right now. If that is correct, then you could fix the issue in two ways.

One option would be to create a second ENI also with a public IP (elastic IP or automatic public IP) and remove the secondary IP from the first ENI.

The second option would be to place a NAT gateway in another subnet where the default route points to the IGW of the VPC. The subnet hosting the EC2 instance would have its default route pointing to the NAT gateway. You would remove the public IP from the instance, and outbound traffic would egress with the public IP of the NAT gateway as its source address.

With the option of using a NAT gateway for outbound access, if inbound access to the instance from the internet is also needed, you'd set up an internet-facing Application Load Balancer (or Network Load Balancer for non-HTTP(S) traffic) in the same subnet with the NAT gateway, so that the public-facing load balancer would also have the IGW as its default gateway. The ALB/NLB would send inbound connections to the private IP address of the EC2 instance configured as the load balancer's target.

AWS EC2 secondary ip outbond traffic

Add your answer

Relevant content