How to collect system logs in EKS Managed Node Group (Amazon Linux 2023) using Fluent Bit?

You're correct that Amazon Linux 2023 handles logging differently from traditional Linux distributions, which can cause challenges when collecting system logs in an EKS Managed Node Group environment. Here's how you can address this situation:

1. Use journald instead of traditional log files: Amazon Linux 2023 uses systemd-journald for log management by default. Instead of trying to collect logs from non-existent files, you should configure Fluent Bit to read from the systemd journal directly.

2. Modify your Fluent Bit ConfigMap: Update your Fluent Bit configuration to use the systemd input plugin. This plugin can read logs from the journal without requiring additional software installation. Here's an example of how you might configure it:

[INPUT]
    Name systemd
    Tag host.*
    Systemd_Filter _SYSTEMD_UNIT=kubelet.service
    Systemd_Filter _SYSTEMD_UNIT=docker.service
    Read_From_Tail On

This configuration will collect logs from the kubelet and docker services. You can adjust the Systemd_Filter options to include other services as needed.

3. Consider using the AWS for Fluent Bit image: AWS provides a Fluent Bit image that's optimized for use with AWS services. This image includes plugins for CloudWatch Logs and other AWS services, which can simplify your configuration.

4. Use a DaemonSet for Fluent Bit: By deploying Fluent Bit as a DaemonSet, you ensure that it runs on all nodes in your cluster, including any new nodes that are added through autoscaling.

5. Leverage EKS Fargate logging (if applicable): If you're using EKS Fargate, you can take advantage of the built-in log router based on Fluent Bit. This can be configured using a specific ConfigMap in the aws-observability namespace.

6. Consider using CloudWatch agent for system-level metrics: While Fluent Bit is great for log collection, you might also want to consider using the CloudWatch agent for collecting system-level metrics from your EKS nodes.

By making these adjustments, you can ensure that Fluent Bit works seamlessly with Amazon Linux 2023 in your EKS environment, collecting the necessary system logs without requiring manual intervention on new nodes. This approach will be more resilient to autoscaling events and provide a consistent logging solution across your entire cluster.
Sources
Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs - Amazon CloudWatch
Unable to locatate syslog and messages in /var/log/ | AWS re:Post
Start AWS Fargate logging for your cluster - Amazon EKS
How to automate application log ingestion from Amazon EKS on Fargate into AWS CloudTrail Lake | AWS Cloud Operations Blog