How to debug Auto-Mode custom NodePool

Question

My EKS cluster is a side project to play with eks- as such, I try to minimize the spend. I updated to eks auto-mode the other day, thinking that being able to remove the default nodepool that karpenter was running in may make up for the extra cost.

I got it working, but the node it spun up was c6 instead of the previous t4. I see that the default is to not use "t" class instances, but several docs about creating custom nodepools, so I gave it a shot. I enabled the "system" built in nodepool (so that the "default" nodeclass and access entry would be created) and then added my own custom one. However, despite trying a bunch of things, no nodes get created.

Error message -
        Failed to schedule pod, incompatible with nodepool "system", daemonset overhead={"pods":"0"}, did not tolerate CriticalAddonsOnly:NoSchedule

Nodepool yaml. You can see it refers to the default nodeclass, and Karpenter reports it as valid (I've tried various things, so I know its possible to get karpenter to say its invalid. However, nothing I've tried actually seems to work. So the real qustion- how do I see the karpenter logs when you're running karpenter for me??

apiVersion: karpenter.sh/v1
    kind: NodePool
    metadata:
      annotations:
        karpenter.sh/nodepool-hash: "4012513481623584108"
        karpenter.sh/nodepool-hash-version: v3
      creationTimestamp: "2024-12-20T22:54:02Z"
      generation: 3
      name: eks-auto-mode
      resourceVersion: "32716"
      uid: e0207f0d-2266-4828-874c-7370e878c692
    spec:
      disruption:
        budgets:
        - nodes: 10%
        consolidateAfter: 0s
        consolidationPolicy: WhenEmptyOrUnderutilized
      template:
        spec:
          expireAfter: 336h
          nodeClassRef:
            group: eks.amazonaws.com
            kind: NodeClass
            name: default
          requirements:
          - key: karpenter.sh/capacity-type
            operator: In
            values:
            - on-demand
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
          - key: eks.amazonaws.com/instance-family
            operator: In
            values:
            - t4g
          terminationGracePeriod: 24h
    status:
      conditions:
      - lastTransitionTime: "2024-12-20T22:54:02Z"
        message: ""
        observedGeneration: 3
        reason: NodeClassReady
        status: "True"
        type: NodeClassReady
      - lastTransitionTime: "2024-12-20T22:54:02Z"
        message: ""
        observedGeneration: 3
        reason: ValidationSucceeded
        status: "True"
        type: ValidationSucceeded
      - lastTransitionTime: "2024-12-21T00:20:15Z"
        message: ""
        observedGeneration: 3
        reason: Ready
        status: "True"
        type: Ready
      resources:
        cpu: "0"
        ephemeral-storage: "0"
        memory: "0"
        nodes: "0"
        pods: "0"

re:Post Agent · Accepted Answer

To debug your custom NodePool in EKS Auto Mode, there are a few things you can check and try:

1. NodePool Configuration: Your NodePool configuration looks generally correct, but there are a couple of points to note:
   - You're specifying the t4g instance family, which is an ARM-based instance type. However, your architecture requirement is set to amd64. This mismatch could be preventing nodes from being created. Consider changing the architecture to arm64 or using an x86-based instance family like t3 or t3a.

2. System NodePool: The error message you're seeing suggests that the pod is trying to schedule on the "system" NodePool, which has a taint for critical add-ons. Make sure your workloads are not accidentally being scheduled on the system NodePool.

3. Logs and Diagnostics: In EKS Auto Mode, AWS manages Karpenter for you, so you don't have direct access to Karpenter logs. However, you can still troubleshoot using:
   - Kubernetes NodeDiagnostic resource to retrieve node logs
   - AWS EC2 CLI command 'get-console-output' to get console output from nodes
   - Check Kubernetes events and node conditions published by the Amazon EKS node monitoring agent

4. IAM and RBAC: Ensure that your AWS IAM and Kubernetes RBAC resources are properly configured for EKS Auto Mode.

5. Resource Requirements: Make sure your pods' resource requests are compatible with the instance types you've specified in your NodePool.

6. Capacity: Check if you have any EC2 instance limits in your AWS account that might be preventing new instances from being launched.

To get more insights:
1. Use 'kubectl describe nodepool eks-auto-mode' to see detailed information about your NodePool.
2. Check Kubernetes events with 'kubectl get events --sort-by=.metadata.creationTimestamp'.
3. Review the status of your pods to see if they're pending or if there are any other issues.

If you're still having trouble, you may need to contact AWS support for more detailed diagnostics, as they have access to the underlying Karpenter logs and infrastructure in EKS Auto Mode.  
**Sources**  
[Troubleshoot EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/auto-troubleshoot.html)  
[Create a Node Pool for EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-node-pool.html)  
[Automate cluster infrastructure with EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/automode.html)

Didier Durand · Answer

Hi,

This page tells you how to access the logs: https://docs.aws.amazon.com/eks/latest/userguide/auto-troubleshoot.html

```
You must use AWS and Kubernetes APIs to troubleshoot nodes. You can:
Use a Kubernetes NodeDiagnostic resource to retrieve node logs.
Use the AWS EC2 CLI command get-console-output to retrieve console output from nodes.
```

This complementary page should also help you:  https://docs.aws.amazon.com/eks/latest/userguide/auto-get-logs.html

Best,

DIdier

How to debug Auto-Mode custom NodePool

Add your answer

Relevant content