EKS Fargate node kubelet version not upgrading to cluster version 1.27


I've been working on updating our EKS Fargate cluster. It was previously on version 1.22 which is no longer in support.

I was able to successfully update the cluster using our terraform setup and update the fargate nodes incrementally through 1.26 and the kubelet on the node would update to match the cluster apiserver version after doing a kubectl rollout restart deployment/<deployment_name>.

However yesterday when I updated the cluster to the latest version available (1.27), the kubelet version on the nodes isn't updating and are still on v1.26.3-eks-f4dc2c0

I need help figuring out what is preventing the fargate nodes from updating their kubelet version to match the apiserver version.

asked 8 months ago518 views
2 Answers

Hello there,

I have replicated this:

  1. Created the Fargate cluster version 1.26,
  2. Rolled out a deployment
  3. Upgraded cluster to 1.27
  4. Restarted the deployment using command : kubectl rollout restart deployment/<deployment_name> -n <namespace>
  5. All the nodes in the namespace updated to 1.27 (v1.27.1-eks-2f008fe)

Please try restarting your deployment in all namespaces. Also from the Kubernetes documentation: https://kubernetes.io/releases/version-skew-policy/#kubelet
The kubelet version 1.26 should not cause any problems since kubelet is supported at 1.27, 1.26, and 1.25

If restarting the deployment does not solve the problem, we would require details that are non-public information. Please open a support case with AWS using the following link: https://support.console.aws.amazon.com/support/home#/case/create

answered 8 months ago

Hi Thubelihle,

I should've followed up on this post, but yesterday, while investigating another issue, I did restart some deployments and noticed the kubelet version updated for those nodes. I restarted the rest of the deployments and all the nodes are now in sync. My guess is that there is a lag in the matching kubelet version being available for the fargate nodes.

I've yet to find any documentation regarding EKS + Fargate + Kubelet version rollouts, but in the future, we're going to wait to upgrade the cluster until we can confirm a matching kubelet version is available. We do understand that kubelet versions have a +/- 1 version support window, but there's no guarantee that future updates won't contain a breaking change that require them to be in sync.

answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions