How do I stream container logs to CloudWatch in Amazon EKS?

Short description

Fluent Bit is the default log solution for CloudWatch Container Insights. It's a best practice to use Fluent Bit instead of Fluentd. Amazon provides a Fluent Bit container image on Amazon Elastic Container Registry (Amazon ECR). For more information, see AWS for Fluent Bit image repositories for Amazon ECS.

To download Fluent Bit, see Getting started with Fluent Bit on the Fluent Bit website. For Fluentd, see Download and install Fluentd on the Fluentd website.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Stream container logs that run in Amazon EKS on an Amazon EC2 cluster

Complete the following steps:

1. Run the following kubectl command to create a namespace that's called amazon-cloudwatch:

   kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

2. Run the following command to create a ConfigMap that's called fluent-bit-cluster-info with the cluster name and AWS Region that you want to send logs to:

   ClusterName=MY-CLUSTER-NAME
   RegionName=MY-CLUSTER-REGION
   FluentBitHttpPort='2020'
   FluentBitReadFromHead='Off'
   [[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
   [[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
   kubectl create configmap fluent-bit-cluster-info \
   --from-literal=cluster.name=${ClusterName} \
   --from-literal=http.server=${FluentBitHttpServer} \
   --from-literal=http.port=${FluentBitHttpPort} \
   --from-literal=read.head=${FluentBitReadFromHead} \
   --from-literal=read.tail=${FluentBitReadFromTail} \
   --from-literal=logs.region=${RegionName} -n amazon-cloudwatch

   Note: Replace MY-CLUSTER-NAME with your cluster's name and MY-CLUSTER-REGION with the Region of your cluster.

3. To deploy Fluent Bit as a DaemonSet to send logs to your Amazon Elastic Compute Cloud (Amazon EC2) cluster, run the following command:

   kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

   After you run the preceding command, Fluent Bit creates the following application, host, and dataplane log groups and sources in CloudWatch:
   The /aws/containerinsights/Cluster_Name/application log source is all log files in the /var/log/containers log group.
   The /aws/containerinsights/Cluster_Name/host log source collects logs from the /var/log/dmesg, /var/log/secure, and /var/log/messages log groups.
   The /aws/containerinsights/Cluster_Name/dataplane log source collects logs from the /var/log/journal for kubelet.service, kubeproxy.service, and docker.service log groups.

4. (Optional) Run the following command to use Fluent Bit on Amazon ECR instead of the default image:

   kubectl patch ds fluent-bit -n amazon-cloudwatch -p \'{"spec":{"template":{"spec":{"containers":[{"name":"fluent-bit","image":"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest"}]}}}}'

5. Install eksctl.
   Note: To run the following command, you must install eksctl.

6. Run the following eksctl command to create AWS Identity and Access Management (IAM) roles for service accounts (IRSA):

   eksctl create iamserviceaccount \
       --name fluent-bit \
       --namespace amazon-cloudwatch \
       --cluster $CLUSTER \
       --attach-policy-arn "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" \
       --approve \
       --override-existing-serviceaccounts
   

Verify the Fluent Bit deployment

Complete the following steps:

1. Run the following kubectl command to list the pod names in amazon-cloudwatch namespace:

   kubectl get pods -n amazon-cloudwatch

   Note: The pod name returns as fluent-bit-*****.
2. Run the following kubectl describe command to check the events output for a specific pod:

   kubectl describe pod POD_NAME -n amazon-cloudwatch

   Note: Replace POD_NAME with the name of your pod.
3. Run the following kubectl command to check the logs:

   kubectl logs pod-name -n amazon-cloudwatch

   Note: Replace POD_NAME with the name of your pod.

Stream container logs that run in Amazon EKS on a Fargate cluster

Amazon EKS on an AWS Fargate cluster has a built-in log router that uses Fluent Bit. AWS automatically runs Fluent Bit on your behalf. For more details, see Start AWS Fargate logging for your cluster.

Complete the following steps:

1. Run the following command to create a dedicated Kubernetes namespace that's called aws-observability:

   cat <<EOF > aws-observability-namespace.yaml
   kind: Namespace
   apiVersion: v1
   metadata:
     name: aws-observability
     labels:
       aws-observability: enabled
   EOF
   
   kubectl apply -f aws-observability-namespace.yaml

2. Run the following command to create a ConfigMap with a Fluent Conf data value to stream container logs to CloudWatch Logs:

   cat <<EOF > aws-logging-cloudwatch-configmap.yaml
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: aws-logging
     namespace: aws-observability
   data:
     output.conf: |
       [OUTPUT]
           Name cloudwatch_logs
           Match   *
           region region-code
           log_group_name fluent-bit-cloudwatch
           log_stream_prefix from-fluent-bit-
           auto_create_group true
           log_key log
   
     parsers.conf: |
       [PARSER]
           Name crio
           Format Regex
           Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
           Time_Key    time
           Time_Format %Y-%m-%dT%H:%M:%S.%L%z
     
     filters.conf: |
        [FILTER]
           Name parser
           Match *
           Key_name log
           Parser crio
   EOF
   
   kubectl apply -f aws-logging-cloudwatch-configmap.yaml

3. Create an AWS managed policy for CloudWatch.
   Note: In the following commands, you attach the IAM policy to the pod execution role that you specified for your Fargate profile. Replace 111122223333 with your AWS account ID, and your-pod-execution-role with the name of your pod execution role.

4. Run the following curl command to download the IAM policy file:

   curl -o permissions.json https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json

5. Run the following create-policy command to create an IAM policy from the policy file:

   aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json

6. Run the following attach-role-policy command to attach the IAM policy to the pod execution role that you specified for your Fargate profile:

   aws iam attach-role-policy \
   --policy-arn arn:aws:iam::111122223333:policy/eks-fargate-logging-policy \
   --role-name your-pod-execution-role

7. Run the following kubectl command to verify that the logging configuration applied to your Fargate pods:

   kubectl describe pod POD-NAME

   Note: Replace POD-NAME with the name of your pod. The output includes Pod events that confirm whether logging is active. For more information, see Troubleshooting.

Clean up the Fluent Bit log streaming deployment

If you deployed Fluent Bit on an EC2 cluster, then run the following kubectl commands to delete the Fluent Bit deployment:

kubectl delete configmap fluent-bit-cluster-info -n amazon-cloudwatch 
kubectl delete -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

If you deployed Fluent Bit on a Fargate cluster, run the following kubectl command to delete the aws-observability namespace. This removes the ConfigMap and stops log streaming:

kubectl delete namespace aws-observability

Note: After you delete the aws-observability namespace, new Fargate pods don't send logs to CloudWatch Logs. Existing pods continue to send logs until you delete and redeploy them. Logs that CloudWatch already received remain available in CloudWatch Logs.

Related information

Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs