Worker Nodes Not Registering in EKS Cluster with Mixed Endpoint Access & Multi-AZ ENIs

Question

I have an EKS cluster configured with both public and private API server endpoints enabled. I am using self-managed worker nodes deployed via Launch Template and Auto Scaling Group on AWS Wavelength in Casablanca. The control plane ENIs are in two different AZs in Paris.

Issue:

When I run:

bash
Copy
Edit
kubectl get nodes
I get:

pgsql
Copy
Edit
No resources found
which indicates the worker nodes are not registering with the control plane.

Details:

1. User Data Script on Worker Nodes
Here is my userdata script:

bash
Copy
Edit
#!/bin/bash
set -ex

echo "Starting EKS bootstrap..." >> /var/log/custom-user-data.log

/etc/eks/bootstrap.sh ${cluster_name} \
  --apiserver-endpoint "${cluster_endpoint}" \
  --b64-cluster-ca "${cluster_ca}" \
  --dns-cluster-ip "${dns_ip}" \
  --container-runtime containerd \
  --kubelet-extra-args "--max-pods=${max_pods}" \
  --use-max-pods false

echo "Bootstrap done." >> /var/log/custom-user-data.log
The log shows that "Starting EKS bootstrap..." is executed, but "Bootstrap done." is never logged.

I wonder if the --apiserver-endpoint parameter should be the public cluster endpoint, or the ENI endpoint (private IPs) of the control plane?

How should I configure this for worker nodes running in Casablanca while control plane ENIs are in Paris?

2. Worker Node Security Group
hcl
Copy
Edit
resource "aws_security_group" "worker_nodes" {
  name        = "${var.cluster_name}-worker-nodes-sg"
  description = "Security group for EKS worker nodes"
  vpc_id      = var.vpc_id

ingress {
    description = "Allow node-to-node communication"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    self        = true
  }

ingress {
    description = "Allow EKS control plane to communicate with nodes"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # I plan to restrict this later to control plane subnets
  }

ingress {
    description = "Control plane to kubelet"
    from_port   = 10250
    to_port     = 10250
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

egress {
    description = "Allow outbound HTTPS traffic to EKS control plane"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  tags = {
    Name = "${var.cluster_name}-worker-sg"
  }
}
Are these security group rules sufficient for the worker nodes to register and communicate with the EKS control plane?
3. Kubernetes aws-auth ConfigMap
I have this configuration for worker node role mapping:

h
Copy
Edit
provider "kubernetes" {
  host                   = var.cluster_endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}

resource "kubernetes_config_map" "aws_auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }

data = {
    mapRoles = yamlencode([
      for arn in var.worker_roles : {
        rolearn  = arn
        username = "system:node:{{EC2PrivateDNSName}}"
        groups   = [
          "system:bootstrappers",
          "system:nodes"
        ]
      }
    ])
  }
}
And the module instantiation:

hcl
Copy
Edit
module "aws_auth" {
  source                 = "../../modules/aws-auth"
  cluster_name           = module.eks.cluster_name
  cluster_endpoint       = module.eks.cluster_endpoint
  cluster_ca_certificate = module.eks.certificate_authority
  worker_roles           = [module.eks_node_group_iam_role.role_arn]
}
The aws-auth configmap applies successfully.

Is this configuration enough to allow worker nodes registration assuming they can reach the control plane?

4. Additional Context
Worker nodes are self-managed on AWS Wavelength in Casablanca.

Control plane ENIs are deployed in two different AZs in Paris.

The cluster has both public and private API server endpoints enabled.

I suspect networking/routing issues due to cross-region setup.

re:Post Agent · Answer

Based on your description, there are several potential issues preventing your worker nodes from registering with the EKS control plane:

1. **Bootstrap Script Configuration**:
For nodes without outbound internet access or in complex networking scenarios like yours (Wavelength in Casablanca connecting to control plane in Paris), the bootstrap parameters are critical. Your script includes the necessary parameters (`--apiserver-endpoint`, `--b64-cluster-ca`), but the bootstrap process isn't completing. For the `--apiserver-endpoint` parameter, you should use the private endpoint if your nodes can route to the control plane ENIs, or the public endpoint if they can't reach the private ENIs directly.

2. **Node Labels Issue**:
For self-managed nodegroups, the bootstrap script needs to include node labels. Without proper node labels, the nodes might join the cluster but not show as "Ready". Add `--node-labels` to your bootstrap script:
```
--kubelet-extra-args "--max-pods=${max_pods} --node-labels=alpha.eksctl.io/nodegroup-name=${cluster_name}"
```

3. **Network Connectivity**:
The fact that "Bootstrap done" is never logged suggests the bootstrap process is failing. This is likely a network connectivity issue between your Wavelength zone (Casablanca) and the control plane ENIs (Paris). Check if:
- Your VPC has proper routing between Wavelength zones and the Paris region
- Security groups allow traffic on ports 443 and 10250 between the worker nodes and control plane ENIs
- Network ACLs aren't blocking the required traffic

4. **Security Group Configuration**:
Your security group configuration allows inbound traffic from anywhere (0.0.0.0/0) on ports 443 and 10250, which is too permissive. You should restrict this to the CIDR blocks of your control plane ENIs. Also, ensure your worker nodes have outbound access to the control plane on port 443.

5. **IAM Role and aws-auth ConfigMap**:
Your aws-auth ConfigMap looks correctly configured. Ensure the IAM role attached to your worker nodes has the necessary EKS permissions, including `eks:DescribeCluster`.

To troubleshoot further:
- Check the kubelet logs on your worker nodes: `journalctl -u kubelet -f`
- Look for errors like "Failed to ensure lease exists", "Unable to register node with API server", or "Failed to contact API server"
- Verify network connectivity from your worker nodes to the control plane endpoints
- Ensure your worker nodes' IAM role has the necessary permissions

For cross-region setups like yours, network connectivity is often the primary challenge. You may need to implement specific routing or VPC peering to ensure your Wavelength nodes can reach the control plane ENIs in Paris.  
**Sources**  
[EKS node creation failed | AWS re:Post](https://repost.aws/questions/QUlvKVjMnSQU2tFRu7ZyJyvw/eks-node-creation-failed)  
[EKS node registration error | AWS re:Post](https://repost.aws/questions/QU5aNGO_EoTiGmPSGszDWsuw/eks-node-registration-error)  
[Announcements - Eksctl User Guide](https://docs.aws.amazon.com/eks/latest/eksctl/announcements.html)  
[Troubleshooting hybrid nodes - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-troubleshooting.html)

Worker Nodes Not Registering in EKS Cluster with Mixed Endpoint Access & Multi-AZ ENIs

Add your answer

Relevant content