How do I resolve the CrashLoopBackOff state for CloudWatch agent and EKS Pod Identity Agent pods?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Check the logs

First, check the CloudWatch agent and EKS Pod Identity Agent logs to gather information about the issue.

To check the CloudWatch agent logs, run the following command:

kubectl logs cloudwatch-agent-pod-name -n namespace

Note: Replace cloudwatch-agent-pod-name with your CloudWatch agent pod name, and namespace with your namespace name.

To check the EKS Pod Identity Agent logs, run the following command:

kubectl logs pod-identity-agent-pod-name -n namespace

Note: Replace pod-identity-agent-pod-name with your EKS Pod Identity Agent pod name, and namespace with your namespace name.

In the command output, look for error messages that show why the pod crashes, such as permission issues, network issues, or configuration issues.

Troubleshoot CloudWatch agent permissions issues

Cannot create provider error

You must use the AWS Identity and Access Management (IAM) service AWS account role to allow Amazon EKS worker nodes to send metrics and logs to CloudWatch. If the IAM role is missing, or incorrectly configured in the required amazon-cloudwatch namespace, then you receive the following error:

"Error: cannot create provider: failed to retrieve credentials: failed to assume role"

To troubleshoot this issue, create a service account role with the cloudwatch-agent name.

To create a custom service account role, run the following command:

eksctl create iamserviceaccount \
--cluster cluster-name \
--namespace amazon-cloudwatch \
--name service-account-name \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--override-existing-serviceaccounts \
--approve

Note: Replace cluster-name with your Amazon EKS cluster name, and service-account-name with the custom service account name.

Then, create a ConfigMap for the CloudWatch agent. If you use a custom service account role, then replace cloudwatch-agent with the service account role name.

Not authorized to perform: sts:AssumeRole errors

If the service account role that the CloudWatch agent uses can't authenticate, then you receive one of the following errors:

"Error: AccessDenied: User: arn:aws:sts::[Account-ID]:assumed-role/[Role-Name]/[Session-Name] is not authorized to perform: sts:AssumeRole on resource [Role-ARN]"

-or-

"Error: AccessDenied: Not authorized to perform sts:AssumeRole"

Make sure that you attached the CloudWatchAgentServerPolicy policy to the service account role. To identify authentication issues, check AWS CloudTrail for PutLogEvents and DescribeLogStreams events. Make sure that you use the cloudwatch-agent service account, or the correct custom service account name in the deployment YAML file.

To check the service account configuration, run the following command:

kubectl get serviceaccount cloudwatch-agent -n amazon-cloudwatch -o yaml

In the output, make sure that the eks.amazonaws.com/role-arn metadata is similar to the following example:

metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ACCOUNT_ID:role/role-name

Service accounts for the CloudWatch agent must have the following rules:

rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "endpoints"]
    verbs: ["list", "watch"]
  - apiGroups: [ "" ]
    resources: [ "services" ]
    verbs: [ "list", "watch" ]
  - apiGroups: ["apps"]
    resources: ["replicasets", "daemonsets", "deployments", "statefulsets"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["nodes/stats", "configmaps", "events"]
    verbs: ["create", "get"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["cwagent-clusterleader"]
    verbs: ["get","update"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get", "list", "watch"]

To check the rules in the service account, run the following command:

kubectl auth can-i --list --as=system:serviceaccount:amazon-cloudwatch:cloudwatch-agent

Troubleshoot EKS Pod Identity Agent permissions issues

Not authorized to perform: eks-auth:AssumeRoleForPodIdentity or Error fetching credentials errors

If the EKS Pod Identity Agent can't assume the Amazon EKS node IAM role, then you receive one of the following errors:

"Error: AccessDenied: User: arn:aws:sts::[Account-ID]:assumed-role/[Role-Name]/[Session-Name] is not authorized to perform: eks-auth:AssumeRoleForPodIdentity on resource [Cluster]"

-or-

"Error: "error","msg":"Error fetching credentials: error getting credentials to cache: unable to fetch credentials from EKS Auth: operation error EKS Auth: AssumeRoleForPodIdentity, https response error StatusCode: 403, RequestID: fc66d1ec-33f1-43b0-a617-df1a52adcb63, AccessDeniedException: ","operation":"AssumeRoleForPodIdentity","request-id":"fc66d1ec-33f1-43b0-a617-df1a52adcb63","service":"EKS Auth""

To troubleshoot this issue, make sure that the IAM policy that you attached to the role allows the AssumeRoleForPodIdentity action. It's a best practice to use the AmazonEKSWorkerNodePolicy AWS managed policy. Or, you can add a custom policy similar to the following example:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"eks-auth:AssumeRoleForPodIdentity"
],
"Resource": "*"
}
]
}

Also, confirm that the service control policies (SCPs) for your organization don't block the AssumeRoleForPodIdentity action.

Important: If you create the pod and service account at the same time as the pod identity association, then you might encounter issues. To resolve the issues, wait at least 10 seconds after you create the role to associate the role to the pod.

Troubleshoot CloudWatch agent network issues

Failed to send logs or Error occurs in PutLogEvents errors

If the CloudWatch Agent can't reach the CloudWatch Logs endpoint, then you receive one of the following errors:

"2023-03-18T12:00:00Z E! [outputs.cloudwatchlogs] Failed to send logs: RequestError: send request failed caused by: Post "https://logs.us-west-2.amazonaws.com/": dial tcp 52.94.76.32:443: i/o timeout"

-or-

"2024-11-22T17:00:48Z E! {"caller":"cwlogs@v0.103.0/cwlog_client.go:135","msg":"cwlog_client: Error occurs in PutLogEvents","kind":"exporter","data_type":"metrics","name":"awsemf/containerinsights","error":"RequestError: send request failed\ncaused by: Post \"https://logs.us-east-1.amazonaws.com/\""

This issue typically occurs because of network restrictions or misconfigured security groups. To troubleshoot, make sure that the nodes have outbound internet access through an internet gateway. For private subnets, confirm that you correctly configured the NAT gateway. Set up the virtual private cloud (VPC) endpoint for CloudWatch services. The VPC endpoint must use the com.amazonaws.Region.logs naming convention, and the endpoint's security group must allow inbound traffic from the node security group.

To confirm that the node security groups allow outbound HTTPS traffic, check for the following configuration:

• Type: HTTPS
• Protocol: TCP
• Port: 443
• Range Destination: 0.0.0.0/0

Troubleshoot EKS Pod Identity Agent network issues

Error retrieving service account token error

The pod security groups must allow outbound HTTP traffic on TCP port 80 to the instance metadata service IP address (169.254.169.254). If they don't, then you receive the following error:

"Error retrieving service account token: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: connection timed out"

To troubleshoot this issue, make sure that the pod security groups use the following configuration:

• Type: HTTPS
• Protocol: TCP
• Port: 80
• Range Destination: 169.254.169.254/32

If your pods use a proxy, then you must add 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6. Update the no_proxy or NO_PROXY environment variables (env) in the pod deployment YAML file.

Example pod deployment YAML file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
spec:
  template:
    spec:
      containers:
        - name: my-container
          image: my-app-image
          env:
            - name: HTTP_PROXY
              value: "http://proxy.example.com:3128"
            - name: HTTPS_PROXY
              value: "http://proxy.example.com:3128"
            - name: NO_PROXY
              value: "localhost,127.0.0.1,169.254.170.23,[fd00:ec2::23]"

Then, to implement your changes, run the following command:

kubectl apply -f deployment.yaml

To check the NO_PROXY setting, run the following command:

kubectl exec -it pod-name -n namespace -- env | grep -i no_proxy

Note: Replace pod-name with the pod name, and namespace with the namespace name.

Make sure that the output looks similar to the following example:

NO_PROXY=localhost,127.0.0.1,169.254.170.23,[fd00:ec2::23]

Troubleshoot CloudWatch agent configuration issues

To identify configuration or resource-related issues for the pod, run the following command to check the pod's detailed status:

kubectl describe pod pod-name -n namespace

Then, take the following troubleshooting steps based on the error that you receive.

Amazon/cloud-watch-agent:1 247345.36b249270" already present on machine error

If you incorrectly configured cloudwatch-agent, then you receive the following error:

"Normal Pulled 14m (x307 over 26h) kubelet Container image "amazon/cloudwatch-agent:1.247345.36b249270" already present on machine Warning BackOff 4m10s (x7130 over 26h) kubelet Back-off restarting failed container aws-cloudwatch-metrics in pod aws-cloudwatch-metrics-4jz88_kube-system(ad6f68f0-7df0-435f-b101-2be05df84eb2)"

To troubleshoot this issue, confirm that you configured the cloudwatch-agent configuration files with the correct AWS Region, logs, and metrics. Also, check for fields that aren't valid or that have missing parameters.

To check if you're using the correct image version, complete the following steps:

1. To list all the pods that run the CloudWatch agent, run the following command:

   kubectl get pods -n amazon-cloudwatch

   Note: If you use a custom namespace, then replace amazon-cloudwatch with the namespace name.

2. To check the image version, run the following command:

   kubectl describe pod pod-name -n amazon-cloudwatch

   Note: Replace pod-name with your pod name. If you use a custom namespace, then replace amazon-cloudwatch with the namespace name.

3. In the output, check the Image value for the version number:

   Containers:
     cloudwatch-agent:
       Image: public.ecr.aws/cloudwatch-agent/cloudwatch-agent:1.300017.0b337

   To see the latest version of CloudWatch agent, see Releases on the GitHub website.

4. If you use an earlier version, then run the following command to update the image version:

   kubectl set image daemonset/aws-cloudwatch-agent \
    -n amazon-cloudwatch \
    cloudwatch-agent=public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest-version

   Note: Replace latest-version with the latest image version.

5. To restart the deployment, run the following command:

   kubectl rollout restart daemonset aws-cloudwatch-agent -n amazon-cloudwatch

If you use the Amazon CloudWatch Observability EKS add-on, then update the add-on to the latest version.

OOM error

If your container has no more available memory, then you receive the following error:

"Warning OOMKilled kubelet Container was killed due to OOM

Warning Failed kubelet Container failed to start: Back-off restarting failed container"

To troubleshoot this issue, check the resource requests and quotas (limits) that you defined in the pod deployment file.

Example pod deployment file:

kubectl get pod cloudwatch-agent-xyz123 -n amazon-cloudwatch -o yaml | grep -A10 'resources:'
If limits are too low, update them in the DaemonSet:
resources:
  requests:
    cpu: 100m
    memory: 200Mi
  limits:
    cpu: 200m
    memory: 400Mi
Apply the changes:
kubectl apply -f cloudwatch-agent-daemonset.yaml

To update the requests and quotas, run the following command:

kubectl edit daemonset aws-cloudwatch-agent -n amazon-cloudwatch
resources:
  requests:
    cpu: 100m
    memory: 200Mi
  limits:
    cpu: 200m
    memory: 400Mi

To implement the changes, run the following command:

kubectl rollout restart daemonset aws-cloudwatch-agent -n amazon-cloudwatch

To make sure that your nodes have enough resources, run the following command:

kubectl top nodes

If the CPU or memory usage is high, then run the following command to scale the cluster:

kubectl scale nodegroup --name nodegroup-name --replicas=new-size

Note: Replace nodegroup-name with your node group name, and new-size with the new node group size.

Troubleshoot EKS Pod Identity Agent configuration issues

Unable to start server error

If you didn't correctly configure EKS Pod Identity Agent, then you might receive the following error:

"{"bind-addr":"[fd00:ec2::23]:80","level":"info","msg":"Starting server...","time":"2024-02-05T17:52:40Z"}{"bind-addr":"[fd00:ec2::23]:80","level":"fatal","msg":"Unable to start server: listen tcp [fd00:ec2::23]:80: socket: address family not supported by protocol","time":"2024-02-05T17:52:40Z"}2024/02/05 17:52:40 running command: exit status 1"

To troubleshoot this issue, make sure that you adhere to the Amazon EKS Pod Identity Agent requirements.

Make sure that you use the latest add-on version to reduce version compatibility issues. To check the latest available version, run the following describe-addon-versions AWS CLI command:

aws eks describe-addon-versions --kubernetes-version 1.31 --addon-name eks-pod-identity-agent 

Note: Replace 1.31 with your cluster version.

To update the addon, run the following update-addon command:

aws eks update-addon --cluster-name my-cluster --addon-name eks-pod-identity-agent --addon-version version-number --resolve-conflicts PRESERVE

Note: Replace my-cluster with your cluster name, and version-number with the add-on version.