NLB Connection issues through IPv6

Hello,

We have an EKS cluster setup with ingress-nginx (the Kubernetes flavor). Configuring the Ingress spawns an NLB (type=network, addressType=Dualstack, scheme=Internet-facing). We changed the NLB to Dualstack mode and added IPv6 addressing to the VPC and subnets. All public subnets are assigned a segment of the IPv6 range provided by AWS at the VPC level (and we can see it reflected on the Network mappings tab on EC2>Load Balances page). Updates were made to the Route 53 DNS records to point the A record as an alias to the NLB, repeated for the AAAA record. In the EKS cluster, we use the IPv4 stack for the EKS containers and services.

We can receive traffic from IPv4 with no issues, but IPv6 traffic seems to fail to connect; we receive IPv6 DNS records for the NLB. No security group is assigned to the NLB; NACLs are open for IPv4 and IPv6 traffic.

Hit list of things checked

• [Pass] Test Connection strictly IPv4
• [ Fail ] Test Connection strictly IPv6 (connection timeout)
• [Pass] VPC IPv6 CIDR Assgined
• [Pass] Subnet Assigned CIDR Block
• [Pass] Subnets have IPv6 Routes propagated
• [Pass] Load Balancer is Dualstack
• [Pass] Load Balancer Network Mapping shows IPv4 and IPv6 addresses (shows "Assigned by AWS" across all 3 AZs)
• [Skip] Load Balancer Security Group permits both IPv4 and IPv6 (N/A, No Security Group Assigned on NLB)
• [Pass] Load Balancer 'DNS Name' is set in Route 53 A Record (yes, Record is set as an Alias to the NLB)
• [Pass] Load Balancer 'DNS Name' is set in Route 53 AAAA Record (yes, Record is set as an Alias to the NLB)
• [Pass] Test DNS to ensure both IPv4 and IPv6 records are being returned (3 of each)
• [Pass] Target Group has available endpoints (they are all IPv4 endpoints; 3 of 4 are healthy. All EKS node ports)
• [Pass] EKS ingress-nginx logs show traffic (Nginx is getting traffic, this is showing only IPv4 source IPs)
• [Pass] NACLs are not blocking traffic related to 80 and 443
• [Pass] EKS Security Groups permit 80 and 443 traffic (yes, explicitly targeting IPv4 IP Version)
• [Pass] Ensure "Deny all IPv6 internet gateway traffic" is unchecked

EKS Service Annotations (for ingress-nginx):

kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  creationTimestamp: "2023-11-17T15:07:20Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.4
  name: ingress-nginx-controller
  namespace: ingress-nginx

I'm not sure what else should be checked to cause IPv6 traffic to timeout.

This is the configuration we're looking for Dual Stack and IPv6-only Amazon VPC Reference Architectures (see Page 7 "6. IPv4 targets for dual stack Internet-facing Network Load Balancer")

Other documentation from AWS