Node in EKS Cluster has status Unknown

same issue facing

When a node in an Amazon EKS cluster shows a status of Unknown, it typically indicates that the Kubernetes control plane has lost communication with the node. This can happen due to various reasons, and there are several steps you can take to troubleshoot the issue:

1. Check node connectivity:

   • Ensure that the node can reach the EKS control plane endpoint. Network issues, security group configurations, or VPC settings could be preventing communication.
   • Verify that the necessary ports are open between the node and the control plane.

2. Investigate kubelet status:

   • SSH into the problematic node and check the kubelet service status using the command: systemctl status kubelet
   • Review kubelet logs for any error messages or indications of why it might not be communicating with the control plane.

3. Examine node conditions:

   • Use kubectl describe node <node-name> to get detailed information about the node's status and conditions.
   • Look for any specific error messages or conditions that might explain the Unknown status.

4. Check AWS-specific components:

   • Ensure that the node has the correct IAM role and permissions to communicate with other AWS services.
   • Verify that the AWS VPC CNI plugin is functioning correctly, as networking issues can cause node communication problems.

5. Investigate system resources:

   • Although you mentioned that CPU and memory usage are below limits, check for any disk space issues or other resource constraints that might be affecting the node's ability to function properly.

6. Review recent changes:

   • If this issue started recently, consider any recent changes to your cluster configuration, security groups, or network settings that might have impacted node communication.

7. Check for known issues:

   • Review the EKS documentation and release notes for any known issues with your specific Kubernetes version or AWS components that might be related to node communication problems.

8. Gather additional logs:

   • Collect logs from key system components such as Docker (or containerd), kubelet, and kube-proxy for further analysis.

If after these steps you're still unable to determine the root cause, you may need to consider the following actions:

• Drain the node and terminate it, allowing a new node to be provisioned in its place.
• If the issue persists across multiple nodes, it may indicate a broader cluster or networking problem that requires further investigation with AWS support.

Remember that in some cases of network disconnections, pods on unreachable nodes may not be immediately evicted, as the Kubernetes control plane has mechanisms to handle temporary disruptions. However, if the issue persists, you'll need to take action to ensure your cluster's health and functionality.
Sources
Kubernetes pod failover through network disconnections - Amazon EKS
Troubleshooting hybrid nodes - Amazon EKS
View the health status of your nodes - Amazon EKS