How do I check if AWS Fargate OS patching deleted my pods or nodes?

Short description

To keep nodes secure, Amazon EKS periodically patches the OS for Fargate nodes. During the patching process, Amazon EKS recycles the nodes to install OS patches. In Fargate OS patching, Amazon EKS uses the eviction API to safely drain the pod and records the API in Amazon EKS audit logs. If Fargate OS patching occurred, then confirm that logs for the eviction API appear in the audit logs. For more information, see the Eviction API on the Kubernetes website.

Resolution

To check that the eviction API appears in the audit logs, use the following example query:

filter @logStream like /^kube-apiserver-audit/ | fields @timestamp, user.username, user.extra.canonicalArn.0, @message
 | sort @timestamp desc
 | filter verb == "create" and objectRef.subresource == 'eviction'
 | filter requestURI like "/api/v1/namespaces/NAMESPACE/pods/pod-name/"

Note: Replace NAMESPACE and pod-name with your values. To narrow down your search window, modify the time window in Amazon CloudWatch.
Example audit log output:

{
        "@logStream": "kube-apiserver-audit",
        "@timestamp": "xxx",
        "@message": {
            "kind": "Event",
            "apiVersion": "audit.k8s.io/v1",
            "level": "RequestResponse",
・・・
            "stage": "ResponseComplete",
            "requestURI": "/api/v1/namespaces/<Namespace>/pods/<Pod_Name>/eviction.."
            "verb": "create",
            "user": {
                "username": "<username>",
・・・
            },
            "userAgent": "<agent>",
            "objectRef": {
                "resource": "pods",
                "namespace": "xxx",
                "name": "xxx",
                "apiVersion": "v1",
                "subresource": "eviction"
            },
            "responseStatus": {
                "metadata": {},
                "status": "Success",
                "code": 201
            },
            "requestObject": {
                "kind": "Eviction",
                "apiVersion": "policy/v1beta1",
                "metadata": {
                    "name": "xxx",
                    "namespace": "xxx",
                    "creationTimestamp": null
                }
            },

Note: To limit the number of pods that are down at the same time when pods are recycled, set pod disruption budgets. For more information, see Specifying a disruption budget for your application on the Kubernetes website.

Related information

API-initiated eviction on the Kubernetes website