Ubuntu nodes failed to join fully private cluster

Question

I have created a fully private cluster and it is working fine (means kubectl, eksctl, and aws commands are working) but there is a problem with the cluster. Whenever I create Amazon Linux 2 node instances, they successfully join the cluster but when I try to create Ubuntu instances then I get the following error message.
```
Instance failed to join the kubernetes cluster,(Service:null, Status Code: 0, Request ID:null)(RequestToken:c912435454-d3d1-2352-542321-4523543243, HandlerErrorCode:GeneralServiceException)

rePost-User-1592414 · Accepted Answer

I have found the solution. As I am working behind the proxy and has no outbound internet access, I need to pass the userdata using OverrideBootstrapCommand.

> For nodegroups that have no outbound internet access, you'll need to supply --apiserver-endpoint and --b64-cluster-ca to the bootstrap script as follows:
 
    overrideBootstrapCommand: |
      #!/bin/bash

source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh

# Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh ${CLUSTER_NAME} --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS}" \
        --apiserver-endpoint ${API_SERVER_URL} --b64-cluster-ca ${B64_CLUSTER_CA}
So when we run the `bootstrap.helper.sh` script then it will automatically find the specified variables in above script and we don't need to do anything.

> Note the --node-labels setting. If this is not defined, the node will join the cluster, but eksctl will ultimately time out on the last step when it's waiting for the nodes to be Ready. It's doing a Kubernetes lookup for nodes that have the label alpha.eksctl.io/nodegroup-name=. This is only true for unmanaged nodegroups.

If you have deployed NAT or any other kind of gateways then The minimum that needs to be used when overriding so eksctl doesn't fail, is labels! eksctl relies on a specific set of labels to be on the node, so it can find them. means there is no need to provide ` --apiserver-endpoint` and `--b64-cluster-ca`

For more details, check this [reference](https://eksctl.io/announcements/nodegroup-override-announcement/)

Venkat Penmetsa · Answer

Hello,

It can happen due to multiple reasons. To understand what is causing the issue, try to SSH into one of the ubuntu instances that failed to join the cluster and check the kubelet status by running `systemctl status kubelet`. If kubelet is in active state, check kubelet logs for any errors by running `journalctl -u kubelet`

This [troubleshooting article](https://aws.amazon.com/premiumsupport/knowledge-center/eks-worker-nodes-cluster) explains about various things to check when your nodes fail to join the cluster. Please check it out.

Ubuntu nodes failed to join fully private cluster

Relevant content