Kubecost AddOn and other remains Degraded after EKS cluster creation in auto-mode

Hello,

'InsufficientNumberOfReplicas' error is generally seen when the required number of required replicas are not running and generally this could be due to nodes are not present to run the replicas. Hence you can check on the points as above mentioned and also if the replicas are started and are fine the status of the addon will be updated to healthy as in auto-mode when you create addon the node will be created afterwards. If this is not helpful you can consider reaching out to us via ticket to check resource details and to further check on any the issue.

The issue you're experiencing with the Kubecost AddOn and other add-ons remaining in a degraded state after EKS cluster creation in auto-mode is likely related to resource allocation or configuration problems. Here are some steps you can take to troubleshoot and potentially resolve the issue:

1. Check the pod status: Use the command kubectl get pods -n <namespace> to see the status of the Kubecost pods. Look for any pods that are in a pending or failed state.

2. Describe the problematic pods: Use kubectl describe pod <pod-name> -n <namespace> to get more detailed information about why the pods are not running properly. This may reveal issues with resource limits, scheduling, or other constraints.

3. Review the add-on configuration: Ensure that the add-on configuration is correct and compatible with EKS auto-mode. You may need to adjust the resource requests and limits for the Kubecost containers.

4. Check node resources: Although you mentioned that you're only using 2% of the resources, verify that there are enough resources available on the nodes to schedule the Kubecost pods. Use kubectl describe nodes to see the allocatable resources and current usage.

5. Examine cluster events: Use kubectl get events --sort-by=.metadata.creationTimestamp to see if there are any relevant events that might explain why the add-on is not deploying correctly.

6. Verify VPC endpoint configuration: Ensure that your VPC endpoints for EKS are correctly configured, as this can impact add-on functionality.

7. Check for conflicting policies or webhooks: Look for any admission webhooks or policies that might be blocking the add-on deployment.

8. Review the EKS add-on logs: Check the Amazon EKS console or use AWS CLI to retrieve the add-on logs for more information on why it's in a degraded state.

9. Consider updating the add-on: If you're using an older version of the Kubecost add-on, try updating to the latest version compatible with your EKS version.

10. Adjust pod priority: If resource constraints are an issue, you may need to adjust the pod priority of the Kubecost agent or scale up your instances to provide more resources.

If these steps don't resolve the issue, you may need to contact AWS support for further assistance, especially if you suspect there might be a compatibility issue between the add-on and EKS auto-mode.
Sources
Runtime coverage and troubleshooting for Amazon EKS clusters - Amazon GuardDuty
Troubleshoot EKS Auto Mode - Amazon EKS
Amazon EKS add-ons - Amazon EKS