ENI Trunking, IPv6 and TCP

I would like to report an issue with IPv6 connectivity in ECS containers running in awsvpc mode. After enabling IPv6 for our VPC and deploying IPv6 networking for our ECS workloads, we noticed that IPv6 downloads from the internet were extremely slow, with speeds less than 20 KB/s.

We conducted several tests and made the following observations:

• Confirmed that all containers/instances had all traffic outbound rule to ::/0. There's no network ACLs.
• Latency and packet loss were normal, ruling out routing or MTU issues.
• We could reproduce the issue on different container images and different AMIs.
• I was able to achieve normal IPv6 TCP performance on an EC2 instance running the same AL2023 AMI, so the issue is isolated to containers running in awsvpc mode (there's no IPv6 support for bridge mode anyway).
• I tested iperf3 in UDP with a target bandwidth of 10 Gbps, the actual received throughput can reach >2 Gbps. This indicates that the issue is specific to TCP.
• Testing with different TCP congestion algorithms did not yield any significant difference in performance.
• Finally, we added an inbound rule for TCP 32768-65535, which immediately restored IPv6 performance to expected levels.

Based on our observations, we suspect that the issue may be related to the EC2 security groups behaving as a “stateless firewall” rather than stateful for IPv6 TCP connections in containers with branch ENIs. This may indicate an issue with the connection tracking.

Any tips on the above will be helpful. Thank you.

Attached three iperf3 logs with identical configuration, except the first one is TCP without any inbound rules, second one is with the IPv6 TCP 32768-65535 inbound rule (which should not be needed for a stateful firewall) and the third one is in UDP mode without any inbound rules.

Iperf3 log without inbound rule.

# iperf3 -c 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8
Connecting to host 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8, port 5201
[  5] local 2001:df0:xxxx:xxxx:5112:37d6:1129:2639 port 45036 connected to 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   323 KBytes  2.64 Mbits/sec    2   8.72 KBytes
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   8.72 KBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   8.72 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   8.72 KBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   8.72 KBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   8.72 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   8.72 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   8.72 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   8.72 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   8.72 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   323 KBytes   265 Kbits/sec    5             sender
[  5]   0.00-10.04  sec  71.7 KBytes  58.5 Kbits/sec                  receiver

iperf Done.

Iperf3 log with IPv6 TCP 32768-65535 inbound rule.

# iperf3 -c 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8
Connecting to host 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8, port 5201
[  5] local 2001:df0:xxxx:xxxx:5112:37d6:1129:2639 port 40924 connected to 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   454 MBytes  3.81 Gbits/sec  233    968 KBytes
[  5]   1.00-2.00   sec   456 MBytes  3.83 Gbits/sec  224   1003 KBytes
[  5]   2.00-3.00   sec   471 MBytes  3.95 Gbits/sec  241    916 KBytes
[  5]   3.00-4.00   sec   452 MBytes  3.80 Gbits/sec  221    846 KBytes
[  5]   4.00-5.00   sec   452 MBytes  3.80 Gbits/sec  214    881 KBytes
[  5]   5.00-6.00   sec   450 MBytes  3.77 Gbits/sec  229   1.04 MBytes
[  5]   6.00-7.00   sec   456 MBytes  3.83 Gbits/sec  214   1.06 MBytes
[  5]   7.00-8.00   sec   471 MBytes  3.95 Gbits/sec  262    907 KBytes
[  5]   8.00-9.00   sec   458 MBytes  3.84 Gbits/sec  243    907 KBytes
[  5]   9.00-10.00  sec   460 MBytes  3.86 Gbits/sec  248    767 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.47 GBytes  3.84 Gbits/sec  2329             sender
[  5]   0.00-10.05  sec  4.47 GBytes  3.82 Gbits/sec                  receiver

iperf Done.

Iperf3 log in UDP mode without inbound rules.

# iperf3 -c  2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8 -u -b 10G
Connecting to host 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8, port 5201
[  5] local 2001:df0:xxxx:xxxx:4a8b:445:8410:65ae port 52683 connected to 2001:df0:xxxx:xxxx:a5c2:75ae:7cee:bde8 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   592 MBytes  4.97 Gbits/sec  69541
[  5]   1.00-2.00   sec   592 MBytes  4.96 Gbits/sec  69498
[  5]   2.00-3.00   sec   592 MBytes  4.96 Gbits/sec  69499
[  5]   3.00-4.00   sec   591 MBytes  4.96 Gbits/sec  69436
[  5]   4.00-5.00   sec   592 MBytes  4.96 Gbits/sec  69499
[  5]   5.00-6.00   sec   592 MBytes  4.96 Gbits/sec  69499
[  5]   6.00-7.00   sec   592 MBytes  4.96 Gbits/sec  69498
[  5]   7.00-8.00   sec   586 MBytes  4.92 Gbits/sec  68844
[  5]   8.00-9.00   sec   591 MBytes  4.96 Gbits/sec  69373
[  5]   9.00-10.00  sec   592 MBytes  4.96 Gbits/sec  69499
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.77 GBytes  4.96 Gbits/sec  0.000 ms  0/694186 (0%)  sender
[  5]   0.00-10.04  sec  2.46 GBytes  2.11 Gbits/sec  0.019 ms  397982/694027 (57%)  receiver

iperf Done.