How do I know that my Amazon EKS control plane was updated?

Short description

Amazon EKS performs continual updates of the API server for high availability so that your cluster doesn't become non-deterministic or unrecoverable. However, API server clients might be affected when the API server instance IP addresses change. It's a best practice for API server clients to manage reconnects. For more information, see Update existing cluster to new Kubernetes version.

Amazon EKS generates new cross AWS account elastic network interfaces in the specified subnets and then removes the previous elastic network interfaces. This operation activates changes in the cluster endpoint IP addresses. If your environment doesn't meet the requirements, then the cross account elastic network interface creation fails. A failed elastic network interface creation can cause the following issues:

• Communication between worker nodes and the control plane.
• Access issues to a private Amazon EKS cluster endpoint.
• The kubectl exec and kubectl logs commands take a long time.
• EKS connectivity latency.

To check that maintenance was performed on your Amazon EKS control plane, check the creation date from the network interface. Or, check the event time from CreateNetworkInterface event history in CloudTrail.

Resolution

To determine whether your Amazon EKS control plane was updated, choose one of the following resolution tasks:

Check cross-account elastic network interfaces on your EKS cluster

To check network interfaces for your clusters, run the following command. To check network interfaces for a specific cluster, replace Amazon EKS* with Amazon EKS example-cluster-name.

aws ec2 describe-network-interfaces --filter Name=description,Values="Amazon EKS*"
aws ec2 describe-network-interfaces --filter Name=description,Values="Amazon EKS example-cluster-name"

-or-

To check whether your Amazon EKS was updated in the console, complete the following steps:

1. Open the Amazon EKS console.
2. Enter your Amazon EKS cluster name in the search bar.
3. Choose the network interfaces with the Interface Type Elastic network interface and the Description Amazon EKS example cluster name.
4. Check the network interface creation date to confirm that the EKS is patched.

Confirm that there are at least two elastic network interfaces for the cluster. The cluster is updated by Amazon EKS when the elastic network interfaces creation date is later than the cluster creation or the user update date.

Check event history

To verify changes in the cross-account elastic network interface for your cluster, review the AWS CloudTrail event history for CreateNetworkInterface. Complete the following steps:

1. Open the CloudTrail console.
2. In the navigation pane, choose Event history.
3. In Lookup attributes, select Event name. Enter CreateNetworkInterface.
4. Check the Event time for entries with Amazon EKS as the User name.
   Note: CloudTrail provides event records for the past 90 days.

Confirm that there are at least two elastic network interfaces for the cluster. If the elastic network interface creation date is later than the cluster creation or the user update date, then the cluster was updated by Amazon EKS.

Retry to create a new cross-account elastic network interface

If there are fewer than two elastic network interfaces, then check the CreateNetworkInterface event in CloudTrail for failures.

The CloudTrail log shows why new cross-account elastic network interfaces failed. To resolve this error, validate that the resources for your cluster meet the VPC requirements.

The following are common reasons for failure:

Subnets and security group

Check that configured subnets and security groups still exist. If subnets and security groups don't exist, then run the following command to update the cluster with new resources in the same VPC:

aws eks update-cluster-config --name <your-cluster> \
  --resources-vpc-config subnetIds=<subnet id>,securityGroupIds=<security id>

Insufficient IP addresses in subnet

To resolve this error, free up at least two IP addresses in the cluster subnets or associate different subnets to your cluster.

Related information

How do I troubleshoot an upgrade fail with my Amazon EKS cluster?

De-mystifying cluster networking for Amazon EKS worker nodes