How to provision GPU nodes in EKS Auto mode type cluster

Question

I have been trying to create GPU Nodes in. EKS auto mode cluster, I have attached the below config for generating the GPU node pool, and using the default nodeclass created in EKS automode. Also if there is a variation of creating our own Node Class it would be very helpful.
yaml config -> 
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
  labels:
    owner: test-cluster
    app.kubernetes.io/managed-by: eks
spec:
  disruption:
    budgets:
    - nodes: 10%
    consolidateAfter: 30s
    consolidationPolicy: WhenEmptyOrUnderutilized
  template:
    metadata: {}
    spec:
      expireAfter: 336h
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: "eks.amazonaws.com/instance-category"
          operator: In
          values: ["g4dn", "g5"]
        - key: "eks.amazonaws.com/instance-gpu-manufacturer"
          operator: In
          values: ["nvidia"]
        - key: "eks.amazonaws.com/instance-gpu-count"
          operator: In
          values: ["1"]
        - key: "eks.amazonaws.com/instance-size"
          operator: In
          values: ["2xlarge"]
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["ap-southeast-1"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
  limits:
    cpu: "1000"
    memory: 1000Gi

re:Post Agent · Answer

To provision GPU nodes in an EKS Auto Mode cluster, you're on the right track with your NodePool configuration. Here's how you can refine your approach and create GPU nodes effectively:

1. NodePool Configuration:
Your provided YAML configuration is a good starting point. It correctly specifies the requirements for GPU instances, including the instance category (g4dn, g5), GPU manufacturer (NVIDIA), GPU count, and instance size. However, you might want to consider the following adjustments:

- Ensure that the specified zone "ap-southeast-1" is correct. Typically, you'd want to specify multiple availability zones for better resilience.
- The CPU and memory limits seem quite high. Adjust these based on your actual needs.

2. NodeClass Configuration:
While you're using the default NodeClass, you can create a custom NodeClass for more specific GPU configurations. Here's how you can create a custom NodeClass:

- Create a YAML file (e.g., gpu-nodeclass.yaml) with your NodeClass configuration.
- Include specific settings for GPU nodes, such as additional EC2 tags or custom IAM roles if needed.
- Apply the configuration to your cluster using kubectl.

3. GPU Support:
EKS Auto Mode includes built-in support for GPU plugins, which simplifies the process of using GPU nodes. This means you don't need to manually install the NVIDIA device plugin.

4. Verification:
After applying your NodePool configuration, verify that the GPU nodes are being provisioned correctly:
- Check the node labels to ensure they have the correct GPU-related labels.
- Run a test pod that requires a GPU to confirm that scheduling works as expected.

5. Considerations:
- Ensure your AWS account has the necessary limits to create the specified GPU instances.
- Be aware of the costs associated with GPU instances, as they are typically more expensive than standard compute instances.

By following these steps and adjusting your configuration as needed, you should be able to successfully provision GPU nodes in your EKS Auto Mode cluster. Remember to monitor your cluster's performance and adjust the NodePool and NodeClass configurations as your workload requirements change.  
**Sources**  
[Create a Node Pool for EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-node-pool.html)  
[Automate cluster infrastructure with EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/automode.html)  
[Create a Node Class for Amazon EKS - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-node-class.html)

Didier Durand · Answer

Hi,

To get a full example of provisioning specific instance nodes with AWS EKS AutoMode, please, read https://devopslearning.medium.com/aws-eks-auto-mode-a-game-changer-or-just-hype-my-unbiased-take-18de17c4484a

Section 2. "Customizing Instance Categories While Creating Nodes in AWS EKS Auto Mode via UI": it uses custom labels and configurations through APIs or CLI commands to influence the instance types within a NodePool.

Best,

Didier

Olawale Olaleye · Answer

In this [blog post](https://aws.amazon.com/blogs/containers/how-to-run-ai-model-inference-with-gpus-on-amazon-eks-auto-mode/) we show you how to swiftly deploy inference workloads on EKS Auto Mode. We also demonstrate key features that streamline GPU management, show best practices for model deployment, and walk through a practical example by deploying open weight models from OpenAI using vLLM.

How to provision GPU nodes in EKS Auto mode type cluster

Add your answer

Relevant content