EKS Auto Mode custom node pool for g instances not getting ephemeral storage

Question

Hello!

I'm trying to deploy a kubernetes workload on EKS using auto mode. My workloads have some specific needs, so i decided to go with a custom nodeclass, and 2 nodepools. Both nodepools use the same nodeclass, one of the nodepools specifies g instances, with some specific taints and tags, and the other specifies general purpose nodes. The general purpose nodes get deployed correctly, the problematic piece is the `ephemeralStorage` property, which is again, correctly set for the general purpose instances. The instance specification is as follows:
```
spec:
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: generic-nodeclass
      requirements:
        - key: "eks.amazonaws.com/instance-family"
          operator: In
          values: ["t2", "m4", "r4", "m5a"]
        - key: "eks.amazonaws.com/instance-size"
          operator: In
          values: [ "small", "medium", "large", "xlarge", "2xlarge", "4xlarge", "8xlarge" ]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
```
However, GPU instances(going to include the requirements section below) only get 20 gigs of ephemeral storage, even though they should be allocating 160 gigs.
GPU node pool reqs:
```
spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: generic-nodeclass
      requirements:
        - key: "eks.amazonaws.com/instance-family"
          operator: In
          values: ["g6", "g4dn"]
        - key: "eks.amazonaws.com/instance-size"
          operator: In
          values: [ "4xlarge", "8xlarge", "12xlarge" ]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
      taints:
        - key: "nvidia.com/gpu"
          value: "true"
          effect: NoSchedule
```

My nodeclass definition is quite barebone
```
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: generic-nodeclass
spec:
  role: ${node_iam_role_name}
  subnetSelectorTerms:
    - tags:
        kubernetes.io/role/internal-elb: "1"
  securityGroupSelectorTerms:
    - tags:
        aws:eks:cluster-name: "${cluster_name}"
  tags:
    karpenter.sh/discovery: "${cluster_name}"
  ephemeralStorage: 
    size: "160Gi"
```

tldr.:

CPU nodepool gets the correct 160 gigs of storage, while gpu instances get only 20, what could be wrong?

kubectl describe nodeclass shows that the ephemeralStorage property is set to 160 gigs, and indicates no issues with the nodeclass.

Todd Neal · Answer

See https://aws.amazon.com/ec2/instance-types/, the g4dn.4xlarge has 225Gi of fast local instance storage. When launching an instance that has local instance storage, EKS Auto Mode will automatically use that local instance storage for ephemeral data (e.g. container images, Pod logs, etc.). When your node class has an ephemeral storage value configured that is less than what the instance already has from local instance storage, it will use a smaller EBS data volume to reduce costs.

Todd Neal

answered a year ago

re:Post Agent · Answer

When using EKS Auto Mode with custom NodePools, there are some specific behaviors to be aware of regarding ephemeral storage configuration, especially for GPU instances.

The issue you're experiencing where GPU instances (g6, g4dn families) only get 20 GiB of ephemeral storage despite specifying 160 GiB in your NodeClass is likely due to how EKS Auto Mode handles different instance types.

In EKS Auto Mode, AWS manages the underlying infrastructure, including the AMIs used for your nodes. For GPU instances, there may be specific configurations or limitations that override your NodeClass ephemeral storage settings.

To troubleshoot this issue:

1. Verify that your NodeClass is correctly referenced in both NodePools by checking the status conditions of your NodePools using `kubectl describe nodepool`.

2. Check Kubernetes events to see if there are any messages related to node provisioning that might explain the discrepancy: `kubectl get events --sort-by=.metadata.creationTimestamp`.

3. Use the Kubernetes NodeDiagnostic resource to retrieve node logs for more information about how the nodes were provisioned.

4. You can also use the AWS EC2 CLI command `get-console-output` to retrieve console output from the nodes to see if there are any clues about the storage configuration.

5. Check if there are any specific requirements or limitations for GPU instances in EKS Auto Mode that might be affecting the ephemeral storage allocation.

It's worth noting that EKS Auto Mode doesn't allow for custom AMIs, so you're limited to the AWS-provided configurations. If the ephemeral storage limitation is a critical issue for your workloads, you might need to consider using standard EKS with self-managed node groups or managed node groups where you have more control over the underlying infrastructure.  
**Sources**  
[How to debug Auto-Mode custom NodePool | AWS re:Post](https://repost.aws/questions/QUWczMGy6MQ4WdctpHA0xkGA/how-to-debug-auto-mode-custom-nodepool)  
[eks auto mode - custom amis | AWS re:Post](https://repost.aws/questions/QUsX411EWrQwyZocI6CFTqxA/eks-auto-mode-custom-amis)  
[Optimizing Karpenter for Resource Constraints: Resolving CPU and Storage Issues in Kubernetes | AWS re:Post](https://repost.aws/articles/ARsI7XCBcsTNaanUtTUbVUbw/optimizing-karpenter-for-resource-constraints-resolving-cpu-and-storage-issues-in-kubernetes)  
[Troubleshoot EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/auto-troubleshoot.html)

EKS Auto Mode custom node pool for g instances not getting ephemeral storage

Add your answer

Relevant content