How do I troubleshoot stuck Pods and namespaces in Amazon EKS clusters?

Short description

After you delete a Pod or namespace in Amazon EKS, the resource enters the Terminating status. If this process doesn't complete, then you must manually resolve the issue.

Pods might experience termination issues for any of the following reasons:

• Finalizers prevent deletion
• Pods fail to respond to termination signals
• PreStop hooks fail or the terminationGracePeriodSeconds service extends too long
• Network disruptions block worker nodes and control plane communication
• High resource usage affects worker nodes

Namespaces might experience termination issues for any of the following reasons:

• Kubernetes can't delete specific resources
• API servers report a False status
• Finalizers prevent deletion

Resolution

Use the following sections to troubleshoot the stuck Pod or namespace.

Troubleshoot the stuck Pods

Complete the following tasks to troubleshoot stuck Pods.

Check the Pod status

Complete the following steps:

1. To check kubelet logs on the affected node, run the following command:

   kubectl get pods -A --field-selector=status.phase==Terminating -o wide

2. Review the NODE column in the output to identify affected nodes. If all stuck Pods are on the same node, then run the following journalctl command to check kubelet logs on the affected node:

   journalctl -u kubelet

3. Restart the kubelet.

Check for Pod finalizers

To check for finalizers, run the following command:

kubectl get pod POD_NAME -n NAMESPACE -o yaml

Note: Replace POD_NAME with the name of your Pod. Replace NAMESPACE with the name of your namespace where your Pod is located.

If finalizers are present, then run one of the following commands:

For simple removal, use the following command:

kubectl patch pod POD_NAME -n NAMESPACE -p '{"metadata":{"finalizers":null}}'

Or, if your environment requires JSON patch operations or you prefer more detailed syntax, then use the following command:

kubectl patch pod POD_NAME -n NAMESPACE --type=json -p '[{"op": "remove", "path": "/metadata/finalizers" }]'

Note: Replace POD_NAME with the name of your Pod. Replace NAMESPACE with the name of your namespace where your Pod is located.

Force delete the Pod

Important: It's a best practice to use this option only if other methods fail to avoid data loss or corruption.

Run the following command to force delete the Pod:

kubectl delete pod POD_NAME -n NAMESPACE --grace-period=0 --force 

Note: Replace POD_NAME with the name of your Pod and replace NAMESPACE with the name of your namespace where your Pod is located.

Troubleshoot the stuck namespaces

Complete the following tasks to troubleshoot the stuck namespaces.

Check namespace status

When a namespace gets stuck in Terminating status, the namespace might contain resources that Kubernetes can't delete. Or, an API service might have a False status.

Run the following command to check the status of your namespace:

kubectl describe namespace NAMESPACE

Note: Replace NAMESPACE with the name of your stuck namespace.

Find and remove remaining resources

If resources remain in the namespace, then you receive the following error message:

"NamespaceContentRemaining: Some resources are remaining: serviceaccounts has 2 resource instances"

To list the remaining resources, run the following command:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n NAMESPACE

Note: Replace NAMESPACE with the name of your namespace.

Then, run the following command to remove the identified resources:

kubectl delete RESOURCE_TYPE RESOURCE_NAME -n NAMESPACE

Note: Replace RESOURCE_TYPE with the type of resource, RESOURCE_NAME with the name of the resource, and replace NAMESPACE with your namespace.

Remove namespace finalizers

If finalizers are still attached, the you receive the following error message:

"NamespaceFinalizersRemaining: Some content in the namespace has finalizers remaining: kubernetes.io/persistent-volume-provisioner in 1 resource instance"

To remove namespace finalizers, you can run a patch command, use a JSON file, or force delete the namespace.

To remove finalizers with a patch operation, run the following command:

kubectl patch namespace NAMESPACE --type=json -p '[{"op": "remove", "path": "/metadata/finalizers" }]'

Note: Replace NAMESPACE with the name of your stuck namespace.

To use a JSON file, complete the following steps:

1. Run the following command to create the namespace configuration file:

   kubectl get namespace TERMINATING_NAMESPACE -o json > tempfile.json

   Note: Replace TERMINATING_NAMESPACE with the name of your stuck namespace.
2. Remove the finalizers array from the spec section of the JSON file.
3. Run the following command to apply the changes:

   kubectl replace --raw "/api/v1/namespaces/TERMINATING_NAMESPACE/finalize" -f ./tempfile.json

   Note: Replace TERMINATING_NAMESPACE with your stuck namespace's name.
4. Verify that the namespace was removed. Run the following command:

   kubectl get namespaces

To force delete the namespace, run the following command:

kubectl delete namespace NAMESPACE --grace-period=0 --force

Note: Replace NAMESPACE with the name of your stuck namespace. It's a best practice to use this option only if other methods fail to avoid data loss or corruption.

Clean up custom resources

If resources remain in the namespace, then run the following command:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n NAMESPACE

Note: Replace NAMESPACE with the name of your namespace.

If you identify custom resources, then run the following command to remove them:

kubectl delete RESOURCE_TYPE RESOURCE_NAME -n NAMESPACE

Note: Replace RESOURCE_TYPE with the type of resource, RESOURCE_NAME with the name of the resource, and replace NAMESPACE with your namespace.

Run the following command to remove custom resource finalizers:

kubectl patch RESOURCE_TYPE RESOURCE_NAME -n NAMESPACE -p '{"metadata":{"finalizers":null}}' --type=merge

Note: Replace RESOURCE_TYPE with the type of resource. Replace RESOURCE_NAME with the name of the resource. Replace NAMESPACE with your namespace.

Then, run the following command to remove CustomResourceDefinitions:

kubectl delete crd CRD_NAME

Note: Replace CRD_NAME with the name of your CustomResourceDefinition. Remove only CustomResourceDefinitions that you no longer need.