How do I configure Amazon EKS worker nodes to clean up the image cache at a specified percent of disk usage?

Short description

To clean up the image cache with Amazon EKS worker nodes, use the following kubelet garbage collection (from the Kubernetes website) arguments:

• The --image-gc-high-threshold argument defines the percent of disk usage that initiates image garbage collection. The default is 85%.
• The --image-gc-low-threshold argument defines the percent of disk usage that image garbage collection tries to free. The default is 80%.

Note: In the following resolution, the kubelet garbage collection arguments clean up the image cache in the worker node when the disk usage reaches 70%. The --image-gc-high-threshold is set to 70%. The --image-gc-low-threshold is set to 50%.

To add kubelet garbage collection arguments to a new worker node, complete the steps in the following sections:

• Add the kubelet garbage collection arguments to the default kubelet-config file in the Amazon Machine Image (AMI) using UserData for a new worker node
• Verify that the new kubelet garbage collection arguments are in the node configz endpoint

To add kubelet garbage collection arguments to an existing worker node, complete the steps in the following sections:

• Add the kubelet garbage collection arguments to an existing worker node
• Verify that the new kubelet garbage collection arguments are in the node configz endpoint

Resolution

Note: The following resolution applies to Amazon EKS-optimized Linux AMIs.

Add the kubelet garbage collection arguments to the default kubelet-config file in the AMI using UserData for a new worker node

1.    Launch an Amazon Elastic Compute Cloud (Amazon EC2) instance.

Use the sed commands to add --image-gc-low-threshold and --image-gc-high-threshold in the kubelet-config.json file in the Amazon EKS-optimized Linux AMI. The sed commands are in the UserData section of Step 3: Configure instance details.

For example:

#!/bin/bash
set -o xtrace

# Inject imageGCHighThresholdPercent value unless it has already been set.
if ! grep -q imageGCHighThresholdPercent /etc/kubernetes/kubelet/kubelet-config.json; 
then 
    sed -i '/"apiVersion*/a \ \ "imageGCHighThresholdPercent": 70,' /etc/kubernetes/kubelet/kubelet-config.json
fi

# Inject imageGCLowThresholdPercent value unless it has already been set.
if ! grep -q imageGCLowThresholdPercent /etc/kubernetes/kubelet/kubelet-config.json; 
then 
    sed -i '/"imageGCHigh*/a \ \ "imageGCLowThresholdPercent": 50,' /etc/kubernetes/kubelet/kubelet-config.json
fi

/etc/eks/bootstrap.sh your-cluster-name

Note: Replace your-cluster-name with your Amazon EKS cluster name. If you're using a different kubelet config file, then update the file name in the sed commands.

2.    Add the worker node to your Amazon EKS cluster.

3.    To verify the changes, follow the steps in the Verify that the new kubelet garbage collection arguments are in the node configz endpoint section.

Add the kubelet garbage collection arguments to an existing worker node

Important: The following steps require that you connect to an existing worker node with SSH and have sudo access. You must complete these steps on all the existing worker nodes in your Amazon EKS cluster.

1.    Connect to an existing worker node using SSH.

2.    Open the /etc/kubernetes/kubelet/kubelet-config.json file in your worker nodes.

If you launched the worker node using EKSCTL, then open /etc/eksctl/kubelet.yaml. See the following example:

sudo vi /etc/kubernetes/kubelet/kubelet-config.json

#WORKER NODES LAUNCHED USING EKSCTL

sudo vi /etc/eksctl/kubelet.yaml

3.    Add the kubelet garbage collection arguments to the kubelet-config.json file or kubelet.yaml file, depending on how you launched your worker nodes. Then, save the file. See the following example:

{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  .
  .
  .
  "imageGCHighThresholdPercent": 70,         ==> Add the argument under the same alignment as the "kind"
  "imageGCLowThresholdPercent": 50,
  "maxPods": ...
}


#WORKER NODES LAUNCHED USING EKSCTL

kind: KubeletConfiguration
kubeReserved:
  cpu: 70m
  ephemeral-storage: 1Gi
  memory: 1843Mi
serverTLSBootstrap: true
imageGCHighThresholdPercent: 70        ==> Add the arguments under the alignment "Kind" in the yaml file
imageGCLowThresholdPercent: 50

4.    To restart the kubelet service in the worker node, run the following command:

sudo service kubelet restart

5.    To verify the changes, follow the steps in the Verify that the new kubelet garbage collection arguments are in the node configz endpoint section.

Verify that the new kubelet garbage collection arguments are in the node configz endpoint

1.    To get the name of your worker nodes, run the following command:

kubectl get nodes

2.    To open a connection to the API server, run the following command:

kubectl proxy

3.    To check the node configz, open a new terminal, and then run the following command:

curl -sSL "http://localhost:8001/api/v1/nodes/node_name/proxy/configz" | python3 -m json.tool

Note: Replace node_name with your node name from the list of nodes retrieved in step 1. If curl and python aren't available, then open the URL in a web browser.

The preceding command returns output from the kubeletconfig. This output includes your settings from the bootstrap.sh file. See the following example:

{
"kubeletconfig": {
  .
  .
  "imageGCHighThresholdPercent": 70,          <=== The new value is set to 70 as given in UserData
  "imageGCLowThresholdPercent": 50,           <=== The new value is set to 50 as given in UserData
  .
  .
}
}

---