AWS VPC CNI - Network policy Architecture and Troubleshooting

Overview of steps to configure Network Policies in EKS

1. Update to the latest Amazon VPC CNI version

2. Enable network policy support in CNI Configuration

   a. If you are using eksctl, you can achieve this with below in the cluster config file

       addons:
          - name: vpc-cni
            version: 1.14.0
            configurationValues: |-
              enableNetworkPolicy: "true"
   

3. Set up the necessary IAM permissions

4. Deploy the network policy controller

   a. When network policies are enabled, the EKS control plane automatically installs the network policy controller. This controller monitors the creation of network policies and configures the necessary eBPF programs on the worker nodes.

5. Define and apply network policies

   a. Below Network Policy denies all ingress traffic by default

     apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: default-deny-all
        spec:
          podSelector: {}
          policyTypes:
          - Ingress
   

6. Deploy application

Amazon VPC CNI Network Policies workflow:

1. NetworkPolicy object creation (EKS Cluster administrator creates network policy)

• The process begins when an EKS Cluster Administrator creates a Kubernetes NetworkPolicy object using kubectl or another Kubernetes management tool. The administrator authenticates using their Kubernetes username and must have appropriate permissions. The administrator's NetworkPolicy manifest is applied to the Kubernetes API server, which creates or updates the NetworkPolicy resource in the Kubernetes data store.
• RBAC verification for Network Policy Controller (Authentication and authorization) -The Kubernetes API server verifies that the Network Policy Controller has proper authorization via the "eks:network-policy-controller" ClusterRole to watch and process NetworkPolicy objects.

2. NetworkPolicy event streaming via Watch API

• The Network Policy Controller (running in EKS Control Plane) receives the NetworkPolicy object event through its established Watch API connection with the Kubernetes API server. This streaming connection allows real-time updates without polling.

• The Network Policy Controller parses the NetworkPolicy rules, which specify which pods can communicate with each other based on labels, namespaces, ports, and protocols.

3. Policy distribution to worker nodes

• The Network Policy Controller distributes the processed policy information to the Network Policy Agent DaemonSet running on each worker node in the cluster.

4. Policy translation by Network Policy Agent

• The Network Policy Agent translates Kubernetes NetworkPolicy definitions into an enforcement model that runs eBPF programs on each node to enforce policy at the kernel level.

6. Agent creates/updates eBPF programs

• The network policy agent takes translated rules creates eBPF programs and updates them dynamically when policies change.These programs operate at kernel level.

7. eBPF program loading

• The VPC CNI Plugin compiles and loads eBPF programs into the Linux kernel on the worker node. These programs will intercept and filter network traffic at the kernel level.
• The eBPF programs update BPF maps (kernel data structures) with policy information. These maps store relationships between pods and their applicable policies, allowed/denied traffic patterns, and other policy metadata.

8 -9 . Pod network traffic interception

• When a pod sends or receives network packets, the eBPF programs intercept this traffic in the kernel before it leaves the host or reaches its destination.
• The eBPF programs query the BPF maps to determine which rules apply to the specific network packet based on source/destination pods, protocols, ports, etc.

10. Traffic filtering decision

• Based on the applicable rules:
  1. If the traffic is allowed by the network policy, the eBPF program permits the packet to proceed to its destination.
  2. If the traffic is denied by the network policy, the eBPF program drops the packet, preventing communication.

Network Policies troubleshooting steps:

1. Review the NetworkPolicies created by customer on EKS Cluster

2. Describe VPC CNI daemonset or Addon and validate the Network Policy mode enabled by the customer - strict vs enforced mode.

3. Apply the same policies on your lab cluster and Test if the network policy is working in lab cluster

   a. If the policies are not working only on customer cluster, check if customer installed any Infra add-ons and security tools like Kyverno, Open Policy Agent that could have Cluster Policies configured and not allowing Network Policy Controller (running on EKS ControlPlane) to access the policies

4. Execute EKS Log collector script and check network-policy-agent.log

5. Network Policy Agent exposes prometheus metrics at port 61680

   a. https://github.com/aws/aws-network-policy-agent/blob/d85dbabad07b2919422c0f95172854da75e3c087/pkg/metrics/metrics.go#L18C16-L18C21

EKS VPC CNI Network policy agent known issues and root causes :

• Error: EKS vpc-cni is generating the networkpolicy logs, given the enable-policy-event-logs is set to false.

  Root Cause: The enable-policy-event-logs setting won't disable the Network Policy (NP) agent logging, but it will only disable the policy "decision" logs. This is documented in the aws-network-policy-agent README.

• Error: Network policy map cleanup

  Root Cause: The issue was caused by a problem with the VPC CNI addon version 1.19.3-eksbuild.1.

• Error: The customer has enabled the network policy feature in the AWS VPC (Virtual Private Cloud) CNI (Container Network Interface) plugin, but they are still seeing issues with the network policies not being applied correctly.

  Root Cause: This appears to be a customer-specific issue with Kyverno. The root cause is that the policyendpoint object was not created in the namespace, which meant the NP controller was unable to create network policy rules for the NP agent to apply.

• Error: When you have network policies enabled in strict mode, pods will come up with default deny policy. Once we have policies applied, traffic will be allowed to the allowed endpoints. When policies are deleted the pod does not go back to default deny state, it goes to default allow state.

  Root Cause: This issue was fixed in the 1.19.3 VPC CNI release, which included the NP agent 1.2.0 release. After the fix, with strict mode, once the policies were removed, the pod would fall back to the default deny state

• Error: The customer is using the Security Groups for Pods (SGFP) feature in EKS, which is known to have some limitations around pod startup latency due to rate limiting in the resource controller.

  Root Cause: The root cause of the issue was identified as API throttling on the CreateNetworkInterface API, which the VPC resource controller uses to create branch ENIs for the pods. The customer's account had a low limit for this API, which was causing delays during high-scale events.

• Errors: FailedScheduling 2m53s (x28 over 137m) default-scheduler 0/5 nodes are available: 5 Insufficient vpc.amazonaws.com/pod-eni. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.

  Root Cause: Assigning Security Groups to pods will increase pod scheduling latency, and HyperPod should consider running its own managed controller like the Node and Certificate controllers.

• Errors: {"level":"info","ts":"2022-02-04T22:24:55.014Z","caller":"entrypoint.sh","msg":"Checking for IPAM connectivity ... "} I0204 22:24:56.095582 12 request.go:621] Throttling request took 1.047064274s, request: GET:https://10.100.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s {"level":"info","ts":"2022-02-04T22:24:57.022Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}panic: runtime error: invalid memory address onil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x39 pc=0x5597a7c2b3f8]

  Root Cause: This issue occurs when a customer installs systemd-udev on AL2023, as the file is re-written with a breaking policy. This can happen when updating to a different releasever that has an updated package or manually updating the package itself.

• Error: {"level":"error","ts":"2025-02-05T20:27:18.669Z","caller":"ebpf/bpf_client.go:578","msg":"failed to find device by name eni9ea69618bf0: %!w(netlink.LinkNotFoundError={0xc000115310})"}

  Root Cause: This issue has been identified and fixed in the latest versions of the AWS VPC CNI network policy agent (v1.2.0).

• Error: Enhanced EKSImageScan CVE Report for CVE https://nvd.nist.gov/vuln/detail/CVE-2025-22868

  Root Cause: The ticket submitter suggests that the scanner can be updated to address the vulnerabilities found in the previous Multus CNI image, version v4.1.4-eksbuild.2_thick. However, since the new version of the Multus CNI image has no vulnerabilities, the next step would be to update the systems to use the new version of the Multus CNI image and the new Network Policy Controller image.

• Error: {"level":"info","ts":"2024-11-25T13:34:24.808Z","logger":"ebpf-client","caller":"events/events.go:193","msg":"Flow Info: ","Src IP":"","Src Port":9096,"Dest IP":"","Dest Port":56830,"Proto":"TCP","Verdict":"DENY"}

  Root Cause: The issue has been resolved in the new version of the Network Policy Controller (NPC).

• Error: Customer has an EKS cluster that they recently upgraded to 1.30 and switched from Calico to AWS VPC CNI. After the switch from Calico to AWS VPC CNI they started having networking issues for pod to pod communication. Previous engineer went through troubleshooting with the customer and noticed when they delete the networking policies that the communication is restablished. When the networking policies are reapplied the communication stops working again.

  Root Cause: The issue was resolved by using port ranges in the network policies, as the maximum number of unique combinations of ports for each protocol in each ingress: or egress: selector in a network policy is 24. The customer was advised to use port ranges to reduce the number of unique ports and avoid the limitation.

• Error: Network policy agent does not support Standalone Pods

  Root Cause: The Network Policy agent currently only supports pods that are deployed as part of a deployment/replicaset. If network policies are applied to standalone pods, there might be some inconsistencies in the behavior, as documented in the aws-network-policy-agent GitHub issue #327

Authors : Sai Teja P, Dashmeet