How to troubleshoot common scaling issues in EKS Auto-Mode

Issue: Service Control Policies (SCPs) Blocking or Restricting Node Provisioning in EKS Auto Mode

Error

User: arn:aws:sts::<account-id>:assumed-role/<EKS-Node-Role>/<session-name> is not authorized to perform: <Service>:<Action> on resource: arn:aws:<Service>:<region>::<Resource>/* with an explicit deny in a service control policy

Causes

1. Explicit Deny in SCPs: An explicit Deny in an AWS Organizations SCP prevents EKS Auto Mode from launching EC2 instances (e.g., ec2:RunInstances).
2. SCPs Requiring Specific Tags: Some SCPs require certain tags on resources, but the default EKS Auto Mode NodeClass does not support custom tags.

Resolution

• For Explicit Deny: Work with your AWS Organizations administrator to remove or adjust the SCP that blocks required actions (e.g., ec2:RunInstances) for the EKS Auto Mode role. Note: Adding an Allow policy does not override an explicit Deny.

---

Issue: Pod Scheduling Failures

1.) Insufficient Resources

Cause: Pods request more CPU, memory, or other resources than available nodes can provide or nodepool exausted of resourced and reached to the limit.
Resolution: Adjust resource requests in pod specs or ensure NodePool configuration allows larger node provisioning.
Example Error:

Warning FailedScheduling 30s (x13 over 60m) default-scheduler 0/5 nodes are available: 1 Insufficient memory. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.

2.) Taints and Tolerations

Cause: Nodes have taints applied without corresponding pod tolerations.
Resolution: Add appropriate tolerations to pod specs or remove/modify unnecessary node taints.

3.) Node Affinity and Node Selectors

Cause: Pod node affinity or selectors restrict scheduling to unavailable nodes.
Resolution: Modify pod node affinity requirements or ensure NodePool configurations can provision suitable nodes.

Cause: Nodepool configured capacity has reached limit

  limits:
    cpu: "100"
    memory: 100Gi

Resolution: Increase the configure resource limits

---

Issue: Incompatible Node Requirements

Cause: Pod node selectors or labels are incompatible with NodePool configurations.

Common Errors::

incompatible requirements, key topology.kubernetes.io/zone, topology.kubernetes.io/zone In [us-east-1a] not in topology.kubernetes.io/zone In [us-east-1b us-east-1c]

Resolution: Use EKS Auto Mode specific labels in your NodePool configuration:

# Pod Manifest
nodeSelector:
  karpenter.sh/nodepool: general-purpose
  topology.kubernetes.io/zone: us-east-1a

---
# NodePool manifest
key: karpenter.sh/nodepool
operator: In
values: 
  general-purpose

Note: EKS Auto Mode uses Karpenter-specific labels like karpenter.sh/nodepool rather than traditional node group labels.

---

Scaling Delays

Issue: Node Provisioning Fails Due to Network Configuration
Cause: EKS Auto Mode provisions nodes in private subnets without proper internet connectivity (NAT Gateway), preventing nodes from communicating with the EKS API server and external services.

• Container image pulling from external registries
• Kubernetes component updates and configurations
• AWS VPC CNI and add-on communications
• EKS API server registration and health checks

Symptoms:

• Nodes are in not ready state and not joining the cluster.
• Multiple node creation attempts
• Pods remain in Pending state for extended periods

Error Examples:

"Unable to register node with API server" err="Post \"https://[CLUSTER-ID].gr7.us-east-1.eks.amazonaws.com/api/v1/nodes\": dial tcp [IP]:443: i/o timeout"

Diagnosis:

1. Check the subnet where EKS Auto Mode is provisioning nodes and verify it has proper internet connectivity
2. Verify the route table has a default route (0.0.0.0/0) pointing to a NAT Gateway

Resolution

1. For Private Subnets: Add a NAT Gateway to the private subnet's route table
2. Route Table Configuration: Ensure the route table includes: 0.0.0.0/0 → nat-gateway-id
3. Alternative: Consider using public subnets for EKS Auto Mode nodes if NAT Gateway is not available

---

Issue: PersistentVolume (PVC) Scheduling Failures during scaling

1.) Incorrect CSI Provisioner Configuration

PersistentVolumeClaims (PVCs) may fail to bind if they are configured with an unsupported or incorrect storage provisioner. This includes:

• Using the wrong CSI provisioner (e.g., ebs.csi.aws.com instead of ebs.csi.eks.amazonaws.com)
• Using deprecated in-tree storage plugins (e.g., AWSElasticBlockStore), which are not supported by EKS Auto Mode

Error Examples:

persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator.

ERROR controller.node_state PersistentVolume source 'AWSElasticBlockStore' uses an in-tree storage plugin which is unsupported by Karpenter and is deprecated by Kubernetes.

Solution: Update your StorageClass and PVC configurations to use the EKS Auto Mode-compatible CSI driver: ebs.csi.eks.amazonaws.comFor more details, see the EBS CSI driver documentation for EKS.

---

Issue: Node Consolidation and Pod Eviction Failures

1.) Scheduling Constraints (Consolidation)

Cause: Cannot consolidate nodes due to pod scheduling constraints.

Resolution:

1. Review and adjust pod affinity/anti-affinity rules
2. Check topology spread constraints for flexibility
3. Ensure resource requests, taints, and tolerations don't prevent consolidation

2.) Pods with Eviction Protection

Cause: Pods with karpenter.sh/do-not-evict annotation cannot be evicted.

Resolution: Remove the annotation to allow eviction.

3.) Pods Protected by PDBs

Cause: Pods governed by Pod Disruption Budgets (PDBs) with strict or restrictive settings may block eviction and prevent node consolidation. If the PDB does not allow for any disruptions at the time of node removal, these pods cannot be evicted.

Resolution: Review and adjust your PDBs to allow at least one disruption if appropriate for your workload, so that pods can be safely evicted during node scaling or consolidation.

---

Additional Resources

• EKS Auto Mode Troubleshooting Guide
• Karpenter Best Practices
• EKS Auto Mode Documentation