I want to check if AWS Fargate operating system (OS) patching deleted my Amazon Elastic Kubernetes Service (Amazon EKS) pods or nodes.
Short description
To keep nodes secure, Amazon EKS periodically patches the OS for Fargate nodes. During the patching process, Amazon EKS recycles the nodes to install OS patches. In Fargate OS patching, Amazon EKS uses the eviction API to safely drain the pod and records the API in Amazon EKS audit logs. If Fargate OS patching occurred, then confirm that logs for the eviction API appear in the audit logs. For more information, see the Eviction API on the Kubernetes website.
Resolution
To check that the eviction API appears in the audit logs, use the following example query:
filter @logStream like /^kube-apiserver-audit/ | fields @timestamp, user.username, user.extra.canonicalArn.0, @message
| sort @timestamp desc
| filter verb == "create" and objectRef.subresource == 'eviction'
| filter requestURI like "/api/v1/namespaces/NAMESPACE/pods/pod-name/"
Note: Replace NAMESPACE and pod-name with your values. To narrow down your search window, modify the time window in Amazon CloudWatch.
Example audit log output:
{
"@logStream": "kube-apiserver-audit",
"@timestamp": "xxx",
"@message": {
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "RequestResponse",
・・・
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces/<Namespace>/pods/<Pod_Name>/eviction.."
"verb": "create",
"user": {
"username": "<username>",
・・・
},
"userAgent": "<agent>",
"objectRef": {
"resource": "pods",
"namespace": "xxx",
"name": "xxx",
"apiVersion": "v1",
"subresource": "eviction"
},
"responseStatus": {
"metadata": {},
"status": "Success",
"code": 201
},
"requestObject": {
"kind": "Eviction",
"apiVersion": "policy/v1beta1",
"metadata": {
"name": "xxx",
"namespace": "xxx",
"creationTimestamp": null
}
},
Note: To limit the number of pods that are down at the same time when pods are recycled, set pod disruption budgets. For more information, see Specifying a disruption budget for your application on the Kubernetes website.
Related information
API-initiated eviction on the Kubernetes website