Elastic Kubernetes Service: NodeCreationFailure "Unhealthy nodes in the kubernetes cluster" and warning "InvalidDiskCapacity 0 on image filesystem" when creating Node Group in EKS

Based on the issues you're experiencing with your EKS node group creation, there are several potential causes for the "NodeCreationFailure: Unhealthy nodes in the kubernetes cluster" error and the "InvalidDiskCapacity invalid capacity 0 on image filesystem" warning.

The most likely issue relates to the bootstrap script in the UserData configuration of your worker nodes. This script is critical for nodes to properly join the EKS cluster, and it must contain the correct EKS cluster name. When this configuration is incorrect, nodes can launch as EC2 instances but fail to register with the Kubernetes cluster.

Here are some specific troubleshooting steps to resolve this issue:

1. Verify the UserData configuration for your worker nodes. Ensure the bootstrap script is present and correctly formatted with your exact cluster name:

#!/bin/bash
/etc/eks/bootstrap.sh your-cluster-name

2. Check if there are any capacity issues with the disk. The "InvalidDiskCapacity invalid capacity 0" warning suggests there might be a problem with the disk allocation or filesystem configuration on the nodes.

3. Review the CloudFormation stack events for the node group creation to identify specific error messages.

4. Ensure your VPC and subnet configurations have the correct DHCP options set, including domain-name and domain-name-servers.

5. Verify that the AWS STS endpoint for your region is enabled for your account.

6. Check that security groups allow necessary traffic between the nodes and the control plane, particularly on port 50051 which is used for node-to-control-plane communication.

7. If you're using a managed node group, verify the node group configuration in the EKS console to ensure the correct cluster name is specified.

8. Try creating a node group in a different region to determine if the issue is region-specific.

9. Check the CNI plugin version compatibility with your Kubernetes version. An incompatible or misconfigured CNI plugin can prevent nodes from becoming ready.

10. Verify that kube-proxy is running correctly by checking its pod status in the kube-system namespace.

If these steps don't resolve the issue, you may need to collect more detailed logs from the worker nodes to identify the specific cause of the failure.
Sources
EKS node creation failed | AWS re:Post
EKS Nodes Fail to Create - but why? It worked yesterday? | AWS re:Post
EKS problem after upgrade from 1.23 | AWS re:Post