How do I troubleshoot logging issues in Amazon EKS?
I want to resolve issues that occur when I turn on logging in Amazon Elastic Kubernetes Service (Amazon EKS) clusters.
Resolution
Before you troubleshoot, make sure that your application writes logs.
Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.
Verify that networking is configured correctly
Verify that networking is configured correctly for your Amazon Virtual Private Cloud (Amazon VPC) subnets and security groups. Check the following:
- If you use an internet gateway, then confirm that it's correctly associated with the route table. Also, confirm that incoming or outgoing network traffic is routing correctly.
- If you use a NAT gateway, then confirm that it's properly configured within a public subnet. Then validate that the associated route table is routing correctly.
- If you have a private cluster with limited internet access, then make sure that your cluster has the com.amazonaws.region.logs VPC interface endpoint with the correct policy attached.
- If you have a service account, then you might have configured the Pods with AWS Identity and Access Management (IAM) roles. If there isn't outbound internet access, then you must use the AWS Security Token Service (AWS STS) VPC endpoint com.amazonaws.region.logs in your VPC.
- You might want to pull images from a private Amazon Elastic Container Registry (Amazon ECR) repository. If you don't have a NAT gateway or outbound internet access, then you need the following VPC endpoints and Amazon Simple Storage Service (Amazon S3) gateway endpoint:
com.amazonaws.region-code.s3
com.amazonaws.region-code.ecr.api
com.amazon.region-code.ecr.dkr - The security group for your VPC endpoint must have an inbound rule that allows traffic from port 443. Also, confirm that the node security group allows outbound traffic to the VPC endpoints.
Verify that the Pods have sufficient permissions
If you use Pods with IAM roles for service accounts, then check that you annotated the service account with the correct IAM role. Then, confirm that you created the IAM OpenID Connect (OIDC) provider for the cluster.
To check the service account's IAM role and permissions, complete the following steps:
-
Describe the service account's IAM role, and then confirm that the following annotation is present. Run the following command:
eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/YOUR-ROLE-NAMENote: ReplaceYOUR-ROLE-NAME with the name of your account's IAM role.
-
Confirm that the IAM role has the CloudWatchAgentServerPolicy attached. Or, confirm that it has a policy with the same permissions. If it doesn't, then make sure that the node role has the CloudWatchAgentServerPolicy attached.
Troubleshoot the Fluent Bit configuration
If you installed the Fluent Bit log router, then confirm that the Pods are running and the system is sending logs to Amazon CloudWatch. If not, then you might have issues with your IAM permissions or networking.
To troubleshoot the Fluent Bit deployment and configuration, verify the following:
To check the Pods in the namespace that you installed Fluent Bit in, run the following command:
kubectl get pods -n NAMESPACE
Note: Replace NAMESPACE with the name of your namespace.
If your nodes have taints, then make sure that the Pods are configured with the tolerations.
If the Pods are in a state other than RUNNING, then run the following command to check the Pod events:
kubectl describe pod POD_NAME -n NAMESPACE
Note: Replace POD_NAME with the name of your Pod, and NAMESPACE with the name of your namespace.
To check the Pod logs for an error, run the following command:
kubectl logs POD_NAME -n NAMESPACE
Note: Replace POD_NAME with the name of your Pod, and NAMESPACE with the name of your namespace.
By default, the log level for Fluent Bit is set to info. If you want more insights, then configure it to debug. For more information, see Configuration file on the Fluent Bit website.
To view your Fluent Bit ConfigMap configurations, run the following command:
kubectl get cm CONFIGMAP_NAME -n NAMESPACE -o yaml
Note: Replace CONFIGMAP_NAME with the name of your ConfigMap, and NAMESPACE with the name of your namespace.
Verify that the node role or role that's associated with the service account has the correct IAM permissions based on the configured destination.
Troubleshoot the CloudWatch Agent
Verify that the node IAM role has the necessary permissions. If you use an IAM role that has a service account role, then make sure that you created an OIDC provider and an IAM role and policy. Verify that the service account is annotated correctly with the appropriate IAM role.
Confirm that the cloudwatch-agent DaemonSet Pods run correctly. If your nodes have taints, then confirm that the Pods are configured with tolerations. To get the list of Pods, run the following command:
kubectl logs POD_NAME -n amazon-cloudwatch
Note: Replace POD_NAME with the name of your Pod.
If the Pods are in a state other than RUNNING, then run check the Pod events. Run the following command:
kubectl describe pod POD_NAME -n amazon-cloudwatch
Note: Replace POD_NAME with the name of your Pod.
Verify that you set the cluster name correctly in the ConfigMap. To check the logs for the Pods, run the following command:
kubectl logs POD_NAME -n amazon-cloudwatch
Note: Replace POD_NAME with the name of your pod.
If you use a public image in the DaemonSet, then make sure that your nodes have internet access. If you use an image from your private repository, then confirm that the required traffic is allowed.
Troubleshoot logging issues on a Fargate cluster
Note: AWS Fargate logging doesn't support dynamic configuration of ConfigMaps. Changes don't apply to existing Pods, so any changes to ConfigMaps apply to new Pods only.
To check that you created a namespace named aws-observability with the aws-observability: enabled label, run the following command:
kubectl get ns aws-observability -o yaml
Confirm that you specified the correct IAM policy in the Pod execution role for your Fargate profile. For example, if you want to send logs to CloudWatch, then make sure that the Pod execution role has the following permissions:
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:CreateLogGroup", "logs:DescribeLogStreams", "logs:PutLogEvents", "logs:PutRetentionPolicy" ], "Resource": "*" }] }
Confirm that you created the ConfigMap correctly in the aws-observability namespace. Verify that the ConfigMap aligns with the rules that Fargate uses to validate fields.
If you're missing the Fluent Bit process logs, then confirm that you set flb_log_cw: "true" in the ConfigMap.
Troubleshoot the CloudWatch Observability add-on in Amazon EKS
If the CloudWatch Observability add-on in Amazon EKS uses an IAM role for service account, then confirm that you created an OIDC provider for your cluster. Then, confirm that you configured the service account with the correct role. The role must have the CloudWatchAgentServerPolicy policy attached.
If you used the AWS Management Console to install the managed add-on, then confirm that the Pods and status of the add-on are active in the console. If the add-on is in a failed status, then check the reason for failure. Similarly, if you used Helm to install the add-on, then confirm that the Helm installation was successful.
If you observe a failure from existing resources, then you might have already installed the required components for the add-on. If you previously installed the CloudWatch agent, then remove it from the cluster before you install the add-on. You must also create a backup of any customizations or modifications that you made to the original CloudWatch agent before you install the add-on. You can reapply the customizations after you install or update the add-on or Helm chart.
Related information
How do I troubleshoot missing container logs for Amazon ECS or Amazon EKS?
How do I turn on Container Insights metrics on an Amazon EKS cluster?
- Topics
- Containers
- Language
- English

Relevant content
- Accepted Answer
asked 3 years ago
asked 2 years ago
- Accepted Answer
asked 5 months ago