How do I configure Amazon EKS worker nodes to clean up the image cache at a specified percent of disk usage?

Short description

To clean up the image cache with Amazon EKS worker nodes, use the following kubelet garbage collection arguments:

• --image-gc-high-threshold defines the percent of disk usage that kubelet starts to run image garbage collection.
  Note: The default for this argument is 85%.
• --image-gc-low-threshold defines the lowest percent of disk usage that kubelet runs kubelet image garbage collection on.
  Note: The default for this argument is 80%.

Based on your use case, add kubelet garbage collection to the default config file or to an existing node. Then, verify that the arguments exist in the node endpoint.

For more information, see Garbage collection of unused containers and images on the Kubernetes website.

Resolution

Note: The following resolution applies to Linux Amazon Machine Images (AMIs) that are optimized for Amazon EKS.

Add the kubelet garbage collection arguments to the default kubelet-config

Complete the following steps:

1. To update the kubelet-config file in the AMI, run the following command:

   #!/bin/bashset -o xtrace
   
   KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
   
   # Inject imageGCHighThresholdPercent value unless it has already been set.
   if ! grep -q imageGCHighThresholdPercent $KUBELET_CONFIG;
   then
   echo "$(jq ".imageGCHighThresholdPercent=70" $KUBELET_CONFIG)" > $KUBELET_CONFIG
   fi
   
   # Inject imageGCLowThresholdPercent value unless it has already been set.
   if ! grep -q imageGCLowThresholdPercent $KUBELET_CONFIG;
   then
   echo "$(jq ".imageGCLowThresholdPercent=60" $KUBELET_CONFIG)" > $KUBELET_CONFIG
   fi
   
   /etc/eks/bootstrap.sh your-cluster-name

   Note: Replace your-cluster-name with your Amazon EKS cluster name. If you don't use the default kubelet config file, then replace kubelet-config.json with your file name. The preceding command sets imageGCHighThresholdPercent to 70%, and imageGCLowThresholdPercent to 60%. As a result, kubelet cleans up the image cache in the worker node when the disk usage reaches 70%. If disk usage is less than 60%, then kubelet doesn't clean up the image cache.
2. Create a worker node group with the launch template.

Add the kubelet garbage collection arguments to an existing worker node

Prerequisites: You must have SSH connection access to an existing worker node and have sudo access.

To add kubelet garbage collection arguments to an existing worker node, complete the following steps:

1. Use SSH to connect to an existing worker node.

2. To open the /etc/kubernetes/kubelet/kubelet-config.json file in your worker nodes, run the following command:

   sudo vi /etc/kubernetes/kubelet/kubelet-config.json

   If you launched the worker node with eksctl, then run the following command to open /etc/eksctl/kubelet.yaml:

   sudo vi /etc/eksctl/kubelet.yaml

3. Add the kubelet garbage collection arguments to the kubelet-config.json file, and then save the file. Example file:

   {  "kind": "KubeletConfiguration",  "apiVersion": "kubelet.config.k8s.io/v1beta1",
     .
     .
     .
     "imageGCHighThresholdPercent": 70,         ==> Add the argument under the same alignment as the "kind"
     "imageGCLowThresholdPercent": 60,
     "maxPods": ...
   }

   Note: The preceding command sets imageGCHighThresholdPercent to 70%, and imageGCLowThresholdPercent to 60%.
   If you launched the worker node with eksctl, then add the kubelet garbage collection arguments to the kubelet.yaml file, and then save the file. Example file:

   kind: KubeletConfiguration
   kubeReserved:
     cpu: 70m
     ephemeral-storage: 1Gi
     memory: 1843Mi
   serverTLSBootstrap: true
   imageGCHighThresholdPercent: 70        ==> Add the arguments under the alignment "Kind" in the yaml file
   imageGCLowThresholdPercent: 60

   Note: The preceding command sets imageGCHighThresholdPercent to 70%, and imageGCLowThresholdPercent to 60%.

4. To restart the kubelet service in the worker node, run the following command:

   sudo service kubelet restart

5. Repeat the previous resolution steps for each existing worker node in your Amazon EKS cluster.

Verify that the new kubelet garbage collection arguments are in the node configz endpoint

Complete the following steps:

1. To get the name of your worker nodes, run the following command:

   kubectl get nodes

2. To open a connection to the API server, run the following command:

   kubectl proxy

3. To check node configz, open a new terminal, and then run the following command:

   curl -sSL "http://localhost:8001/api/v1/nodes/node_name/proxy/configz" | python3 -m json.tool

   Note: Replace node_name with your node name. If you can't use curl or Python, then open the http://localhost:8001/api/v1/nodes/node_name/proxy/configz URL in a web browser.
   Example output:

   {"kubeletconfig": {  .
     .
     "imageGCHighThresholdPercent": 70,          <=== The new value is set to 70 as given in UserData
     "imageGCLowThresholdPercent": 60,           <=== The new value is set to 50 as given in UserData
     .
     .
   }
   }

   Note: The preceding command output shows your settings for the bootstrap.sh file from the kubeletconfig.