How to ensure system logs are consistently collected and sent to CloudWatch from all Amazon EKS Managed Nodes (Amazon Linux 2023), including new nodes from autoscaling?

Question

My Amazon EKS cluster uses a Managed Node Group with Amazon Linux 2023, and I’ve deployed Fluent Bit as a DaemonSet on all nodes to collect system logs and forward them to CloudWatch.

My Fluent Bit configuration includes the following input plugin in the ConfigMap:
[INPUT]
    Name                      systemd
    Tag                          host.*
    Path                        /var/log/journal
    Read_From_Tail    On

I’ve mounted the necessary systemd journal paths using the following volumeMounts and volumes in the DaemonSet:
volumeMounts:
  - name: journal
    mountPath: /var/log/journal
    readOnly: true
  - name: runlogjournal
    mountPath: /run/log/journal
    readOnly: true
  - name: machineid
    mountPath: /etc/machine-id
    readOnly: true

volumes:
  - name: journal
    hostPath:
      path: /var/log/journal
  - name: runlogjournal
    hostPath:
      path: /run/log/journal
  - name: machineid
    hostPath:
      path: /etc/machine-id

We're currently using the Fluent Bit image:
906394416424.dkr.ecr.eu-west-1.amazonaws.com/aws-for-fluent-bit:2.32.0.20231205
We had also tried the public image:
public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
However, in both cases, the logs are not reaching CloudWatch.

I do not want to manually enable systemd journal forwarding on each node using commands, as my node group is managed and auto-scaled — manual changes would be lost on new nodes.

Question: What is the best way to ensure system logs (e.g., journal logs from systemd) are consistently collected and sent to CloudWatch from all EKS Managed Nodes (Amazon Linux 2023), including newly launched ones via autoscaling?
Thanks in advance for your help!

re:Post Agent · Answer

To ensure system logs are consistently collected and sent to CloudWatch from all Amazon EKS Managed Nodes running Amazon Linux 2023, including new nodes from autoscaling, you need to make several adjustments to your current setup.

First, your approach of using Fluent Bit as a DaemonSet is correct, as this ensures the log collector runs on all nodes, including any new ones added through autoscaling. However, your configuration needs some adjustments to work properly with Amazon Linux 2023, which uses systemd-journald for log management by default.

Here are the recommended steps:

1. **Update your Fluent Bit configuration**: Your current systemd input configuration is close, but may need refinement. Ensure you have specific Systemd_Filter options to target the services you want to monitor:

```
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=docker.service
Read_From_Tail On
```

2. **Use launch templates with your managed node group**: Create a launch template that includes user data scripts to ensure proper CloudWatch agent configuration. This approach ensures that any new node created through autoscaling will have the correct configuration automatically.

3. **Ensure proper IAM permissions**: Make sure the IAM role attached to your EKS nodes includes the `CloudWatchAgentServerPolicy` and `AmazonSSMManagedInstanceCore` AWS managed policies to allow proper log forwarding.

4. **Verify volume mounts**: Your volume mounts look correct for accessing the systemd journal, but ensure that all paths exist on the Amazon Linux 2023 instances.

5. **Consider enabling Container Insights**: This AWS solution automatically collects metrics and logs from your EKS cluster and can simplify your logging setup.

6. **Implement log rotation**: Set up log rotation to manage log file sizes and prevent disk space issues.

7. **Monitor log ingestion**: Set up alerts for log ingestion failures to ensure your logging pipeline is working correctly.

If you're still experiencing issues with logs not reaching CloudWatch, check the Fluent Bit pods' logs for any errors or configuration issues. Also verify that your CloudWatch output plugin is correctly configured with the proper region and log group settings.

By implementing these recommendations, you should be able to ensure consistent system log collection across all your EKS nodes, including those added through autoscaling.  
**Sources**  
[How to collect system logs in EKS Managed Node Group (Amazon Linux 2023) using Fluent Bit? | AWS re:Post](https://repost.aws/questions/QU81gvurAcRi6ezXs5k2rsYQ/how-to-collect-system-logs-in-eks-managed-node-group-amazon-linux-2023-using-fluent-bit)  
[Logging for Amazon EKS - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/kubernetes-eks-logging.html)  
[Types of logging in Amazon EKS - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/amazon-eks-observability-best-practices/log-types.html)  
[Best practices for logging in Amazon EKS - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/amazon-eks-observability-best-practices/logging-best-practices.html)  
[Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html)

How to ensure system logs are consistently collected and sent to CloudWatch from all Amazon EKS Managed Nodes (Amazon Linux 2023), including new nodes from autoscaling?

Add your answer

Relevant content