Skip to content

How to Select Only Trunk-Compatible Instance Types in a Karpenter NodePool

0

Hello,

I’m currently configuring Karpenter with Security Groups for Pods enabled. In this setup, only instance types that support ENI trunking can host pods using secondary security groups.

However, I haven’t found an easy way to make Karpenter automatically select only trunk-compatible instance types in a NodePool. At the moment, it seems the only option is to manually maintain an allow-list of compatible instance types, but this approach is hard to scale and error-prone.

Is there a native mechanism (e.g., constraint, label, or instance capability selector) to have Karpenter automatically filter instance types based on their trunking compatibility?

For context:

I’m aware that the amazon-vpc-resource-controller-k8s defines trunking capability per instance type (see pkg/aws/vpc/limits.go ).

My goal is to reuse this logic (or similar metadata) so that Karpenter can dynamically exclude non-trunk-compatible instance types when provisioning nodes for workloads that rely on security groups for pods.

Could you please confirm whether this feature exists or if there’s a planned enhancement to expose trunking support as a filterable capability in Karpenter?

Thank you very much for your time and clarification.

3issa Al Msirdi

asked a year ago367 views

1 Answer
0

When configuring Karpenter with Security Groups for Pods enabled, you need to ensure that only instance types supporting ENI trunking are selected. While there isn't a direct trunking-specific selector in Karpenter, you can use a combination of instance category and generation requirements to target compatible instances.

In your Karpenter NodePool configuration, you can specify requirements that will help filter for more modern instance types that typically support trunking:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: example
spec:
template:
spec:
requirements:
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]  # Select compute, memory, or general purpose optimized
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["2"]  # Select newer generation instances
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: example

This approach filters for newer generation instances that are more likely to support trunking capabilities. You can further refine this by adjusting the instance categories and minimum generation number.

For more precise control, you could use the EC2 instance selector tool to identify specific trunk-compatible instances and then explicitly list them in your configuration. While this is more manual, it gives you certainty about the instances being selected.

Monitoring your node resource usage regularly with tools like kubectl top nodes will help you verify that the selected instances are meeting your workload requirements while supporting the trunking capabilities needed for Security Groups for Pods.
Sources
Kubernetes Data Plane - Amazon EKS
Karpenter Support - Eksctl User Guide
Optimizing Karpenter for Resource Constraints: Resolving CPU and Storage Issues in Kubernetes | AWS re:Post

answered a year ago

  • I've found this way more efficient to work with spot instances

    apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: trunk-compatible-family-exclusion spec:

    Define your limits (e.g., maximum total CPU or memory for the nodepool)

    limits: cpu: "1000"

    Specify the consolidation configuration

    consolidation: enabled: true

    template: spec: # Define the common provisioner/nodepool requirements requirements: # Exclude all the instance families that are not ENI trunk compatible - key: karpenter.k8s.aws/instance-family operator: NotIn values: - c1 - c3 - c4 - d2 - f1 - g3 - h1 - i2 - i3 - i4i - m1 - m2 - m3 - m4 - p2 - p3 - r3 - r4 - t1 - t2 - t3 - t3a - t4g # Specific U-series prefixes - u-6tb1 - u-9tb1 - u-12tb1 - u-18tb1 - u-24tb1 - x1 - x1e

        # Standard requirements for OS and Architecture
        - key: kubernetes.io/os
          operator: In
          values:
            - linux
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64 # or arm64, or both
    

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.