EKS Node Group - CloudFormation

Based on the error message and details provided, there are several potential reasons why your EKS node instances are failing to join the cluster:

1. UserData Bootstrap Script Issue: The most common cause of this error is that the bootstrap script in the UserData of the worker node doesn't contain the correct EKS cluster name. The bootstrap script must include the proper command to join the specific cluster you've created. In a CloudFormation template, verify that the UserData section correctly references your cluster name.

2. IAM Role Configuration: While you mentioned that the Node Group uses a role with required policies, there might be issues with how it's being used. Make sure:

• The IAM role specified in the NodeRole property has all necessary permissions
• You're not using the same role that was used to create the cluster
• If you're using a launch template, ensure you haven't specified an IamInstanceProfile in the launch template (this would conflict with the NodeRole)

3. Network Configuration: In a private VPC with private-only endpoint access, ensure:

• Your VPC has the correct DHCP options configured
• All necessary VPC endpoints are properly set up (you mentioned EKS, EC2, and SSM, but you might also need endpoints for ECR, S3, and others depending on your setup)
• Security groups allow all required traffic between nodes and control plane

4. aws-auth ConfigMap: The aws-auth ConfigMap is typically managed automatically for managed node groups, but you can verify it by:

• Connecting to your cluster using kubectl
• Running kubectl get configmap aws-auth -n kube-system -o yaml
• Checking that your node IAM role ARN is properly mapped in the mapRoles section

5. Capacity Issues: Sometimes specific Availability Zones may not have sufficient capacity to support new nodes.

To troubleshoot further:

• Check the CloudFormation stack events for more detailed error messages
• Review the EC2 instance logs (you can access them via SSM Session Manager if you have the SSM endpoint configured)
• Verify that the AMI being used is compatible with your EKS cluster version
• Try creating a node group with different instance types or in different subnets

If you're using a ready-made template without changes, it's possible that some parameters need to be adjusted for your specific environment, particularly the cluster name in the bootstrap script.
Sources
EKS node creation failed | AWS re:Post
AWS::EKS::Nodegroup - AWS CloudFormation
EKS Nodes Fail to Create - but why? It worked yesterday? | AWS re:Post
Create a managed node group for your cluster - Amazon EKS

EKS Node Group Troubleshooting Guide:

1. Most Common Causes in Private Only Mode: a. Missing or misconfigured VPC endpoints: Required endpoints:

   • com.amazonaws.[region].ecr.api
   • com.amazonaws.[region].ecr.dkr
   • com.amazonaws.[region].s3 b. Missing routes in private subnet route tables c. Incorrect aws-auth ConfigMap configuration

2. Verifying the aws-auth ConfigMap: Command: kubectl get configmap aws-auth -n kube-system -o yaml

   Expected entry: mapRoles:

   • groups:
     • system:bootstrappers
     • system:nodes rolearn: arn:aws:iam::<account-id>:role/<your-node-role-name> username: system:node:{{EC2PrivateDNSName}}

3. Additional Verification Steps: a. Check Node Group IAM Role permissions: Commands: - aws iam get-role --role-name <your-node-role-name> - aws iam list-attached-role-policies --role-name <your-node-role-name>

   Required policies: - AmazonEKSWorkerNodePolicy - AmazonEKS_CNI_Policy - AmazonEC2ContainerRegistryReadOnly

   b. Verify VPC Endpoint Access: Command: aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<your-vpc-id>

   c. Check Node Bootstrap Logs: Step 1: Get AMI ID aws ssm get-parameter --name /aws/service/eks/optimized-ami/<eks-version>/amazon-linux-2/recommended/image_id --region <region>

   Step 2: Connect to the instance using Session Manager and run: sudo cat /var/log/user-data.log sudo cat /var/log/cloud-init-output.log