IPv6 Neighbor Discovery does not work with Prefix Delegation without an extra IPv6 address?

1

I'm facing a problem with Assign Prefix feature in EC2. It seems that delegated prefixes do not work for local communication without an additional (maybe out-of-prefix?) address.

Setup

  • single VPC subnet with IPv4 and IPv6 enabled
  • Security Groups allow all outgoing traffic (0.0.0.0/0 and ::/0) and also all inbound traffic from the whole VPC (whole VPC subnets for IPv4 and IPv6)
  • two instances
  • each instance has a single network interface in the same subnet
  • each instance has a single IPv4 address automatically assigned
  • each instance has one IPv6 /80 prefix delegated to the instance using https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-prefix-eni.html
  • each instance is manually configured with 256 IPv6 addresses from the beginning of the delegated prefix - so all addresses starting from <prefix>::1 to <prefix>:100 are assigned.

Observed behavior

  • instances can reach anything outside their subnet via IPv4 - good
  • instances can also reach anything outside their subnet - via IPv6 - using any of 256 addresses as source address - also good
  • instances can talk to each other via IPv4 - also good
  • instance can't contact each other using IPv6 - all attempts at communication fail at Neighbor Discovery Protocol attempts

Symptoms

$ ping -I <prefix1>::11 <prefix2>::22
PING <prefix2>::22(<prefix2>::22) from <prefix1>::11 : 56 data bytes
^C
--- <prefix2>::22 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2062ms

Running tcpdump on both instances suggests that ICMPv6 messages for Neighbor Discovery Protocol (NDP) do not pass between instances in subnet if the instance does not have assigned an IPv6 address (as opposed to assigning whole prefix).

Am I missing some configuration somewhere? Or is this a bug?

tcpdump output - broken configuration

Setup with prefix delegation only (no explicit IP address allocated in AWS API).

Instance #1 - attempts to ping the other instance using it's IP address (ping source):

listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:52:40.955401 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56
15:52:42.389921 ens5  Out IP6 i-0ce004b0204e4cdee > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2600:1f18:634c:d17f:3447::22, length 32
15:52:43.428126 ens5  Out IP6 i-0ce004b0204e4cdee > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2600:1f18:634c:d17f:3447::22, length 32
15:52:44.452143 ens5  Out IP6 i-0ce004b0204e4cdee > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2600:1f18:634c:d17f:3447::22, length 32
15:52:45.476212 lo    In  IP6 i-0ce004b0204e4cdee > i-0ce004b0204e4cdee: ICMP6, destination unreachable, unreachable address 2600:1f18:634c:d17f:3447::22, length 112
15:52:45.476221 lo    In  IP6 i-0ce004b0204e4cdee > i-0ce004b0204e4cdee: ICMP6, destination unreachable, unreachable address 2600:1f18:634c:d17f:3447::22, length 92
15:52:45.476258 lo    In  IP6 i-0ce004b0204e4cdee > i-0ce004b0204e4cdee: ICMP6, destination unreachable, unreachable address 2600:1f18:634c:d17f:3447::22, length 112
15:52:45.476293 lo    In  IP6 i-0ce004b0204e4cdee > i-0ce004b0204e4cdee: ICMP6, destination unreachable, unreachable address 2600:1f18:634c:d17f:3447::22, length 112
15:52:50.955371 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56

Instance #2 - ping target - should see ICMPv6 ND messages but they are just not there - check the timestamps:

listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:52:39.036744 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56
15:52:49.036683 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56
15:52:59.036613 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56

Workaround?

It seems that local IPv6 traffic starts working if we manually add an additional IPv6 address (not prefix) using https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#assign-ipv6-address .

Configuration change - add an auto-generated IPv6 address to both instances when they are running and retry ping. The web console says it does does Action=AssignIpv6Addresses&Version=2016-11-15&Ipv6AddressCount=1.

tcpdump output - working config - prefix delegation + 1 extra IPv6 address

Instance #1 - attempts to ping the other instance using it's IP address (ping source):

listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:58:00.955500 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56
15:58:02.854944 ens5  Out IP6 i-0ce004b0204e4cdee > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has i-06fd416c51d2541fe, length 32
15:58:02.854977 ens5  In  IP6 i-06fd416c51d2541fe > i-0ce004b0204e4cdee: ICMP6, neighbor advertisement, tgt is i-06fd416c51d2541fe, length 32
15:58:02.854987 ens5  Out IP6 i-0ce004b0204e4cdee > i-06fd416c51d2541fe: ICMP6, echo request, id 2, seq 1, length 64
15:58:02.855495 ens5  In  IP6 i-06fd416c51d2541fe > i-0ce004b0204e4cdee: ICMP6, echo reply, id 2, seq 1, length 64
15:58:03.032678 ens5  Out IP6 i-0ce004b0204e4cdee > ff02::1:ff00:100: ICMP6, neighbor solicitation, who has i-06fd416c51d2541fe, length 32
15:58:03.032701 ens5  In  IP6 i-06fd416c51d2541fe > i-0ce004b0204e4cdee: ICMP6, neighbor advertisement, tgt is i-06fd416c51d2541fe, length 32

Instance #2 - ping target - receives ICMPv6 ND messages and responds as it should:

listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:57:59.036203 ens5  M   IP6 _gateway > ff02::1: ICMP6, router advertisement, length 56
15:58:02.852442 ens5  In  IP6 i-0ce004b0204e4cdee > i-06fd416c51d2541fe: ICMP6, echo request, id 2, seq 1, length 64
15:58:02.852479 ens5  Out IP6 i-06fd416c51d2541fe > ff02::1:ff00:11: ICMP6, neighbor solicitation, who has i-0ce004b0204e4cdee, length 32
15:58:02.852507 ens5  In  IP6 i-0ce004b0204e4cdee > i-06fd416c51d2541fe: ICMP6, neighbor advertisement, tgt is i-0ce004b0204e4cdee, length 32
15:58:02.852521 ens5  Out IP6 i-06fd416c51d2541fe > i-0ce004b0204e4cdee: ICMP6, echo reply, id 2, seq 1, length 64
15:58:03.310085 ens5  Out IP6 i-06fd416c51d2541fe > ff02::1:ff00:100: ICMP6, neighbor solicitation, who has i-0ce004b0204e4cdee, length 32
15:58:03.310112 ens5  In  IP6 i-0ce004b0204e4cdee > i-06fd416c51d2541fe: ICMP6, neighbor advertisement, tgt is i-0ce004b0204e4cdee, length 32

... and ping now works:

3 packets transmitted, 3 received, 0% packet loss, time 2045ms
  • Additional complication

    EC2 API does not support creating instances with IPv6 prefix and IPv6 address at the same time. API call which specifies both parameters errors out with:

    An error occurred (InvalidParameterValue) when calling the RunInstances operation: You can and can only specify one of ipv6-addresses or ipv6-addresses-count ipv6-prefix and ipv6-prefix-count

    It seems like we need to add a second workaround on top of the initial workaround to get this to work.

  • What does the route table look like on the instance?

pspacek
asked a year ago65 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions