Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

EKS auto Custom NodePools Network Plugin not initializing

0

Hi, I've tried multiple times to get started with EKS auto, when I create the predefined Nodepools and NodeClass everything works as a charm, but I want to manage Nodepools myself, which also includes managing your own NodeClasses.

However, when setting everything up with the documentation by my side everything seems to be going well:

  • EKS can create EC2 nodes
  • EC2s join the eks cluster using the EKS-Auto role However, the node fails to reach the Ready status

From the Node Yaml output showd that the Network Plugin is not ready

Conditions:
  Type                    Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                    ------  -----------------                 ------------------                ------                       -------
  MemoryPressure          False   Wed, 16 Jul 2025 12:48:37 +0200   Wed, 16 Jul 2025 12:33:10 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure            False   Wed, 16 Jul 2025 12:48:37 +0200   Wed, 16 Jul 2025 12:33:10 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure             False   Wed, 16 Jul 2025 12:48:37 +0200   Wed, 16 Jul 2025 12:33:10 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                   False   Wed, 16 Jul 2025 12:48:37 +0200   Wed, 16 Jul 2025 12:33:10 +0200   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  StorageReady            True    Wed, 16 Jul 2025 12:50:00 +0200   Wed, 16 Jul 2025 12:50:00 +0200   DiskIsReady                  Monitoring for the Disk system is active
  NetworkingReady         True    Wed, 16 Jul 2025 12:50:00 +0200   Wed, 16 Jul 2025 12:50:00 +0200   NetworkingIsReady            Monitoring for the Networking system is active
  KernelReady             True    Wed, 16 Jul 2025 12:50:00 +0200   Wed, 16 Jul 2025 12:50:00 +0200   KernelIsReady                Monitoring for the Kernel system is active
  ContainerRuntimeReady   True    Wed, 16 Jul 2025 12:50:00 +0200   Wed, 16 Jul 2025 12:50:00 +0200   ContainerRuntimeIsReady      Monitoring for the ContainerRuntime system is active

Going into the node as follows:

kubectl get nodes
kubectl debug node/i-nodename -it --profile=sysadmin --image=public.ecr.aws/amazonlinux/amazonlinux:2023 -- bash
yum install -y util-linux-core

I can view the logs of the network policy agent as follows:

bash-5.2# nsenter -t 1 -m cat /var/log/aws-routed-eni/network-policy-agent.log
{"level":"info","ts":"2025-07-15T07:49:52.361Z","logger":"grpcLogger","caller":"config/controller_config.go:89","msg":"Trying to establish GRPC connection to local IPAM process"}
{"level":"info","ts":"2025-07-15T07:51:27.664Z","logger":"grpcLogger","caller":"config/controller_config.go:89","msg":"Trying to establish GRPC connection to local IPAM process"}

So there seems to be something not going well on the Networking CNI side.

The regular way to EKS networking is to install the VPC CNI addon, but according to the EKS auto documentation, EKS Auto Manages the plugins for you (source https://docs.aws.amazon.com/eks/latest/userguide/automode.html#_automated_components)

Weirdly enough, I tried to create a Built-in node pools from the management console by going to the EKS cluster and clicking Create at the Built In Node Pool tab and I created the Workpool Built-in nodepool.

The deployment of the Built-in nodepool fails, (I suspect because it conflicts with my custom NodeClass that I created), however, this fixed my existing node:

bash-5.2# nsenter -t 1 -m cat /var/log/aws-routed-eni/network-policy-agent.log
{"level":"info","ts":"2025-07-15T09:20:42.306Z","logger":"grpcLogger","caller":"config/controller_config.go:89","msg":"Trying to establish GRPC connection to local IPAM process"}
{"level":"info","ts":"2025-07-15T09:22:17.573Z","logger":"grpcLogger","caller":"config/controller_config.go:89","msg":"Trying to establish GRPC connection to local IPAM process"}
{"level":"info","ts":"2025-07-15T09:23:52.969Z","logger":"grpcLogger","caller":"config/controller_config.go:89","msg":"Trying to establish GRPC connection to local IPAM process"}
{"level":"info","ts":"2025-07-15T09:24:56.660Z","logger":"grpcLogger","caller":"config/controller_config.go:105","msg":"connected to ipamd grpc endpoint and get response for settings","enablePolicyEventLogs":false,"enableIPv6":false}
{"level":"info","ts":"2025-07-15T09:24:56.663Z","logger":"controllers.policyEndpoints","caller":"controllers/policyendpoints_controller.go:88","msg":"ConntrackTTL","cleanupPeriod":300}

The network agent initialized and the pod enters a ready state.

I deleted the built in node pool afterwards, but everything stayed okay, and I could keep creating new nodes.

I have no idea what happens under the hood. But apparently trying to add a built-in node pool from the console fixed something.

I recreated the cluster to check whether this would fail on a new cluster and I see the same behavior as before, where the network plugin could not initialize.

How do I know why my custom node does not initialize? Who manages the VPC CNI addon? Do the plugins even get added to the cluster?

I am very stuck and I don't know whether I should even stick to EKS auto or go back to Karpenter.

Topics
Containers
Tags
Amazon EKS Auto Mode
Language
English

asked a year ago654 views

3 Answers
1

To debug your EKS Auto Mode custom NodePool networking issue, I can provide some insights on what's happening and how to troubleshoot it.

In EKS Auto Mode, AWS manages the networking components for you, including the container networking interface (CNI). The traditional VPC CNI addon configuration options don't apply to EKS Auto Mode as it uses a new networking capability that handles node and pod networking.

When you create a cluster with custom NodePools and NodeClasses (instead of using the built-in ones), you need to ensure your NodeClass is properly configured for networking. The networking configuration is specified in the NodeClass Kubernetes object, not through the traditional VPC CNI addon.

What likely happened when you created the built-in node pool from the console is that it triggered the proper initialization of the networking components that were missing in your custom setup. Even though the built-in node pool deployment failed, it seems to have properly configured the networking components that your custom nodes could then use.

To troubleshoot this issue:

  1. Check your NodeClaims to see if any are in a non-ready state:
kubectl get nodeclaim
  1. For any NodeClaims that show Ready = False, examine the details:
kubectl describe nodeclaim <node_claim>

Look under the Status section for issues preventing the node from joining.

  1. Use the NodeDiagnostic resource to retrieve node logs for further investigation.

  2. You can also use the AWS EC2 CLI command get-console-output to retrieve console output from your nodes.

  3. Review your custom NodeClass configuration to ensure it has the proper networking settings.

When using custom NodePools and NodeClasses in EKS Auto Mode, you need to ensure your NodeClass includes the necessary networking configurations. The fact that creating a built-in node pool fixed your issue suggests that your custom NodeClass might be missing some required networking configurations.

Remember that in EKS Auto Mode, AWS manages the networking plugins for you, but you need to provide the proper configuration through the NodeClass resource. The traditional approach of installing the VPC CNI addon separately is not applicable in EKS Auto Mode.
Sources
How to debug Auto-Mode custom NodePool | AWS re:Post
Troubleshoot EKS Auto Mode - Amazon EKS
Learn about VPC Networking and Load Balancing in EKS Auto Mode - Amazon EKS

answered a year ago

0
Accepted Answer

I got it fixed with AWS business support: Apparently I forgot to couple the correct policy to the node_iam_role in the EKS access entry. Adding this policy to the node_iam_role access entry fixed it: policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAutoNodePolicy"

answered a year ago

0

Thanks for the quick reply!

Other noted I forgot to add: Infra is rolled out using this module https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest , where we then create the custom nodeclaim and nodepools using parameterized Kubernetes manifests.

Running Kubernetes version 1.33 and eks platform version eks.6

I did add some networking specific things in my ec2nodeclass Will paste the relevant details for the nodeclass, most just grabbed default values:

apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
....

  networkPolicy: DefaultAllow
  networkPolicyEventLogs: Disabled
  role: (role)
  securityGroupSelectorTerms:
  - tags:
      my-tags
  snatPolicy: Random
  subnetSelectorTerms:
  - tags:
      Environment: dev
      Tier: private
  tags:
    Name: karpenter.sh/ec2nodeclass/default
    environment: dev
status:
  conditions:
  - lastTransitionTime: "2025-07-16T09:05:56Z"
    message: ""
    observedGeneration: 1
    reason: SubnetsReady
    status: "True"
    type: SubnetsReady
  - lastTransitionTime: "2025-07-16T09:05:56Z"
    message: ""
    observedGeneration: 1
    reason: SecurityGroupsReady
    status: "True"
    type: SecurityGroupsReady
  - lastTransitionTime: "2025-07-16T09:05:57Z"
    message: ""
    observedGeneration: 1
    reason: InstanceProfileReady
    status: "True"
    type: InstanceProfileReady
  - lastTransitionTime: "2025-07-16T09:05:57Z"
    message: ""
    observedGeneration: 1
    reason: CapacityReservationsReady
    status: "True"
    type: CapacityReservationsReady
  - lastTransitionTime: "2025-07-16T09:06:04Z"
    message: ""
    observedGeneration: 1
    reason: ValidationSucceeded
    status: "True"
    type: ValidationSucceeded
  - lastTransitionTime: "2025-07-16T09:06:04Z"
    message: ""
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  instanceProfile: (instance-profile)
  securityGroups:
  - id: sg-redacted
    name: (node-security-group)
  subnets:
  - id: (redacted1)
     zone: 
     zoneiD:
  - id: (redacted2)
  - id: (redacted3)

Everything for the nodeclass (and the nodepools) shows that the resources are ready

Nodeclaims

Name:         default-2xqdf
Namespace:    
API Version:  karpenter.sh/v1
Kind:         NodeClaim
(truncated)
Status:
  Allocatable:
    Cpu:                  1830m
    Ephemeral - Storage:  89Gi
    Memory:               3061Mi
    Pods:                 27
  Capacity:
    Cpu:                  2
    Ephemeral - Storage:  100Gi
    Memory:               3788Mi
    Pods:                 27
  Conditions:
    Last Transition Time:  2025-07-17T07:07:26Z
    Message:               Node status is NotReady
    Observed Generation:   1
    Reason:                NodeNotReady
    Status:                Unknown
    Type:                  Initialized
    Last Transition Time:  2025-07-17T07:07:28Z
    Message:               
    Observed Generation:   1
    Reason:                Launched
    Status:                True
    Type:                  Launched
    Last Transition Time:  2025-07-17T07:07:42Z
    Message:               
    Observed Generation:   1
    Reason:                Registered
    Status:                True
    Type:                  Registered
    Last Transition Time:  2025-07-17T07:07:26Z
    Message:               Initialized=Unknown
    Observed Generation:   1
    Reason:                ReconcilingDependents
    Status:                Unknown
    Type:                  Ready
  Image ID:                ami-0f73043f0d34fff21
  Node Name:               i-redacted
  Provider ID:             aws:///eu-west-1b/i-redacted
Events:
  Type    Reason             Age                  From       Message
  ----    ------             ----                 ----       -------
  Normal  Launched           6m39s                karpenter  Status condition transitioned, Type: Launched, Status: Unknown -> True, Reason: Launched
  Normal  DisruptionBlocked  6m37s                karpenter  Nodeclaim does not have an associated node
  Normal  Registered         6m25s                karpenter  Status condition transitioned, Type: Registered, Status: Unknown -> True, Reason: Registered
  Normal  DisruptionBlocked  36s (x3 over 4m37s)  karpenter  Node isn't initialized

Nodeclaim already shows not-ready and node not initialized

Looking at nodediagnostics:

❯ kubectl get nodediagnostics.eks.amazonaws.com
No resources found

Back to the Node again:

Name:               i-redacted
Conditions:
  Type                    Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                    ------  -----------------                 ------------------                ------                       -------
  MemoryPressure          False   Thu, 17 Jul 2025 09:12:59 +0200   Thu, 17 Jul 2025 09:07:41 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure            False   Thu, 17 Jul 2025 09:12:59 +0200   Thu, 17 Jul 2025 09:07:41 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure             False   Thu, 17 Jul 2025 09:12:59 +0200   Thu, 17 Jul 2025 09:07:41 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                   False   Thu, 17 Jul 2025 09:12:59 +0200   Thu, 17 Jul 2025 09:07:41 +0200   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  NetworkingReady         True    Thu, 17 Jul 2025 09:13:58 +0200   Thu, 17 Jul 2025 09:13:58 +0200   NetworkingIsReady            Monitoring for the Networking system is active
  KernelReady             True    Thu, 17 Jul 2025 09:13:58 +0200   Thu, 17 Jul 2025 09:13:58 +0200   KernelIsReady                Monitoring for the Kernel system is active
  ContainerRuntimeReady   True    Thu, 17 Jul 2025 09:13:58 +0200   Thu, 17 Jul 2025 09:13:58 +0200   ContainerRuntimeIsReady      Monitoring for the ContainerRuntime system is active
  StorageReady            True    Thu, 17 Jul 2025 09:13:58 +0200   Thu, 17 Jul 2025 09:13:58 +0200   DiskIsReady                  Monitoring for the Disk system is active
Events:
  Type     Reason                   Age                    From                   Message
  ----     ------                   ----                   ----                   -------
  Normal   Starting                 7m48s                  kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity      7m48s                  kubelet                invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  7m47s (x2 over 7m48s)  kubelet                Node i-0183f09fb5a8d455a status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    7m47s (x2 over 7m48s)  kubelet                Node i-0183f09fb5a8d455a status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     7m47s (x2 over 7m48s)  kubelet                Node i-0183f09fb5a8d455a status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  7m47s                  kubelet                Updated Node Allocatable limit across pods
  Normal   Synced                   7m46s                  cloud-node-controller  Node synced successfully
  Normal   RegisteredNode           7m43s                  node-controller        Node i-0183f09fb5a8d455a event: Registered Node i-0183f09fb5a8d455a in Controller
  Normal   DisruptionBlocked        97s (x4 over 7m38s)    karpenter              Node isn't initialized

Still note the error Ready False Thu, 17 Jul 2025 09:12:59 +0200 Thu, 17 Jul 2025 09:07:41 +0200 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized for the node

I don't see what the additional configuration for the nodeclass would change, because I don't see that much specific settings to tweak, and most of them defaulted to an Allow state anyway.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content