How do I stream container logs to CloudWatch in Amazon EKS?

Short description

Because Fluent Bit is the default log solution for CloudWatch Container Insights, it's a best practice to use Fluent Bit instead of Fluentd. Amazon provides a Fluent Bit container image on Amazon Elastic Container Registry (Amazon ECR). For more information, see AWS for Fluent Bit image repositories for Amazon ECS.

To download Fluent Bit, see Getting started with Fluent Bit on the Fluent Bit website. For Fluentd, see Download Fluentd on the Fluentd website.

When you set up Fluent Bit as a DaemonSet to send logs to CloudWatch, Fluent Bit creates the following log groups and sources:

• /aws/containerinsights/Cluster_Name/application: Log source is all log files in /var/log/containers.
• /aws/containerinsights/Cluster_Name/host: Log source is logs from /var/log/dmesg, /var/log/secure, and /var/log/messages.
• /aws/containerinsights/Cluster_Name/dataplane: Log source is the logs in /var/log/journal for kubelet.service, kubeproxy.service, and docker.service.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Stream container logs that run in Amazon EKS on an Amazon EC2 cluster

Complete the following steps:

1. To create a namespace that's called amazon-cloudwatch, run the following command:

   kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

2. To create a ConfigMap that's called fluent-bit-cluster-info that includes the cluster name and AWS Region that you want to send logs to, run the following command:

   ClusterName=my-cluster-name
   RegionName=my-cluster-region
   FluentBitHttpPort='2020'
   FluentBitReadFromHead='Off'
   [[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
   [[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
   kubectl create configmap fluent-bit-cluster-info \
   --from-literal=cluster.name=${ClusterName} \
   --from-literal=http.server=${FluentBitHttpServer} \
   --from-literal=http.port=${FluentBitHttpPort} \
   --from-literal=read.head=${FluentBitReadFromHead} \
   --from-literal=read.tail=${FluentBitReadFromTail} \
   --from-literal=logs.region=${RegionName} -n amazon-cloudwatch

   Note: Replace my-cluster-name and my-cluster-region with your cluster's name and Region.

3. To use Fluent Bit to send logs to an Amazon Elastic Compute Cloud (Amazon EC2) cluster, run the following DaemonSet command:

   kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

4. (Optional) To use Fluent Bit on Amazon ECR, run the following aws-for-fluent-bit DaemonSet command:

   kubectl patch ds fluent-bit -n amazon-cloudwatch -p \'{"spec":{"template":{"spec":{"containers":[{"name":"fluent-bit","image":"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest"}]}}}}'

5. To create AWS Identity and Access Management roles for service accounts (IRSA), run the following eksctl command:

   eksctl create iamserviceaccount \
       --name fluent-bit \
       --namespace amazon-cloudwatch \
       --cluster $CLUSTER \
       --attach-policy-arn "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" \
       --approve \
       --override-existing-serviceaccounts

   For more information, see Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs.

Troubleshoot Fluent Bit deployment

Complete the following steps:

1. To list the pod names in amazon-cloudwatch namespace, run the following command: 

   kubectl get pods -n amazon-cloudwatch

   Note: The pod name returns as fluent-bit-*****.

2. To check the events output, run the following command:

   kubectl describe pod POD_NAME -n amazon-cloudwatch

   Note: Replace POD_NAME with the name of your pod. 

3. To check the logs, run the following command:

   kubectl logs pod-name -n amazon-cloudwatch

   Note: Replace POD_NAME with the name of your pod.

Delete Fluent Bit deployment

To delete Fluent Bit deployment, run the following commands:

kubectl delete configmap fluent-bit-cluster-info -n amazon-cloudwatch 
kubectl delete -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

Stream container logs that run in Amazon EKS on an Fargate cluster

Amazon EKS on an AWS Fargate cluster has a built-in log router that's based on Fluent Bit. AWS automatically runs Fluent Bit for you. For more details, see Start AWS Fargate logging for your cluster.

To stream containers logs to CloudWatch Logs, complete the following steps:

1. To create a dedicated Kubernetes namespace that's called aws-observability, run the following command:

   cat <<EOF > aws-observability-namespace.yaml
   kind: Namespace
   apiVersion: v1
   metadata:
     name: aws-observability
     labels:
       aws-observability: enabled
   EOF
   
   kubectl apply -f aws-observability-namespace.yaml

2. To create a ConfigMap with a Fluent Conf data value to stream container logs to CloudWatch Logs, run the following command:

   cat <<EOF > aws-logging-cloudwatch-configmap.yaml
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: aws-logging
     namespace: aws-observability
   data:
     output.conf: |
       [OUTPUT]
           Name cloudwatch_logs
           Match   *
           region region-code
           log_group_name fluent-bit-cloudwatch
           log_stream_prefix from-fluent-bit-
           auto_create_group true
           log_key log
   
     parsers.conf: |
       [PARSER]
           Name crio
           Format Regex
           Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
           Time_Key    time
           Time_Format %Y-%m-%dT%H:%M:%S.%L%z
     
     filters.conf: |
        [FILTER]
           Name parser
           Match *
           Key_name log
           Parser crio
   EOF
   
   kubectl apply -f aws-logging-cloudwatch-configmap.yaml

3. Create an AWS managed policy for CloudWatch. Then, attach the IAM policy to the pod execution role that you specified for your Fargate profile.

   To download the IAM policy file, run the following command:

   curl -o permissions.json https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json

   To create an IAM policy from the policy file, run the following create-policy command:

   aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json

   To attach the IAM policy to the pod execution role that you specified for your Fargate profile, run the following attach-role-policy command:

   aws iam attach-role-policy \
   --policy-arn arn:aws:iam::111122223333:policy/eks-fargate-logging-policy \
   --role-name your-pod-execution-role

   Note: Replace 111122223333 with your account ID.

   For more information, see Troubleshooting.

Turn off streaming logs for EKS on Fargate pods

To turn off streaming logs for your EKS on Fargate pods, run the following command:

kubectl delete namespace aws-observability

Delete pods, and then redeploy them after you delete the aws-observability namespace.