How do I troubleshoot setup issues when I integrate Fluent Bit with Container Insights for Amazon EKS?

Short description

Fluent Bit is a lightweight log processor and forwarder that you use to collect container logs in Amazon CloudWatch.

It's a best practice to use Fluent Bit instead of Fluentd (a legacy open-source program) because of its low resource footprint and the Use_Kubelet feature. For more information, see Turn on the Use_Kubelet feature for large clusters.

To allow Fluent Bit to deliver container logs to Amazon CloudWatch Logs, you must grant AWS Identity and Access Management (IAM) permissions to Fluent Bit. With Amazon EKS, there are two ways to grant IAM permissions:

• Attach a policy to the IAM role of your worker nodes.
• Use an IAM service account role.

If you grant IAM permissions to Fluent Bit, then Fluent Bit is able to perform the following actions:

• logs:DescribeLogGroups
• logs:DescribeLogStreams
• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents

Common issues include:

• Fluent Bit pods crash.
• Fluent Bit doesn't send logs to CloudWatch.
• Fluent Bit pods return CreateContainerConfigError.

Resolution

Set up an IAM role for service account

Create an IAM role for the cloudwatch-agent service account for the amazon-cloudwatch namespace with the AWS managed policy CloudWatchAgentServerPolicy.

1.    Run the following commands to set up environment variables:

export CLUSTER="clustername"
export AWS_REGION="awsregion"
export AWS_ACCOUNT="awsaccountid"

Note: Replace "clustername", "awsregion", and "awsaccountid" (including the quotation marks) with your own cluster name, AWS Region, and account ID.

2.    Run the following eksctl command:

eksctl create iamserviceaccount \
    --name cloudwatch-agent \
    --namespace amazon-cloudwatch \
    --cluster $CLUSTER \
    --attach-policy-arn "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" \
    --approve \
    --override-existing-serviceaccounts

For more information on Container Insights prerequisites, see Verify prerequisites.

Set up the CloudWatch agent to collect cluster metrics and turn on Container Insights

1.    To deploy Container Insights using the quick start, run the following command:

ClusterName="my-cluster-name"
RegionName="my-cluster-region"
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || 
FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-
container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-
monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/
{{cluster_name}}/'${ClusterName}'/;s/
{{region_name}}/'${RegionName}'/;s/
{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/
{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/
{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/
{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f -

Note: Replace "my-cluster-name" and "my-cluster-region" (including the quotation marks) with your own cluster name and AWS Region.

The preceding command creates a namespace, ClusterRole, ClusterRoleBinding, and ConfigMap for the CloudWatch agent and Fluent Bit.

2.    After that command runs, run the following command to create the Fluent Bit service account:

eksctl create iamserviceaccount \
    --name fluent-bit \
    --namespace amazon-cloudwatch \
    --cluster $CLUSTER \
    --attach-policy-arn "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" \
    --approve \
    --override-existing-serviceaccounts

3.    Run the following command to validate that the CloudWatch agent deploys:

kubectl get pods -n amazon-cloudwatch

4.    When complete, the CloudWatch agent creates the log group /aws/containerinsights/Cluster_Name/performance. Then, the CloudWatch agent sends the performance log events to the log group.

Troubleshooting

Fluent Bit pods crash

1.    Check for error messages in the Fluent Bit pod logs. Run these commands to look for events from Fluent Bit pods:

kubectl -n amazon-cloudwatch logs -l k8s-app=fluent-bit 

kubectl -n amazon-cloudwatch describe pod fluent_pod pod_name

2.    Verify that cluster-info (the Fluent Bit configuration file stored in ConfigMap) is accurate and has no syntax mistakes. Make sure that all cluster name and Region values are set. For more information, see amazon-cloudwatch-container-insights on the GitHub website.

Fluent Bit doesn't send logs to CloudWatch

1.    Verify that the output plugin is set up properly in the Fluent Bit configuration file. To check if there are any data ship errors, use the following command to check the Fluent Bit pod logs:

kubectl -n amazon-cloudwatch logs fluent_pod_name

2.    Make sure that the Fluent Bit pods have the necessary IAM permissions to stream logs to CloudWatch. Your Amazon EKS worker nodes submit metrics and logs to CloudWatch because of the CloudWatchAgentServerPolicy enforced by IAM. There are two ways to grant the necessary IAM permissions:

• Add a policy to the worker nodes' IAM roles.
• Create an IAM role for the cluster's service accounts, and affix the policy to it.

See the section Set up an IAM role for service account for more information.

Fluent Bit pods stuck in CreateContainerConfigError

If the pod status is CreateContainerConfigError, then run the following command to get the exact error message:

kubectl describe pod pod_name -n amazon-cloudwatch

In the Events section from the output of the command, look for an error message like the following:

Error syncing pod ("fluent-bit-xxxxxxx"), skipping: failed to "StartContainer" with CreateContain
erConfigError: "configmap \"fluent-bit-config\" not found"

If you see this error message, then it's likely that you didn't create the ConfigMap for Fluent Bit (fluent-bit-config). Follow the installation steps again to be sure to create the ConfigMap.

---

Related information

Set up the CloudWatch agent to collect cluster metrics

Quick Start with the CloudWatch agent and Fluent Bit

Enable debug logging (GitHub website)