By using AWS re:Post, you agree to the Terms of Use

Fluent Bit Logs, Kinesis vs OpenSearch (ElasticSearch) Directly

0

I have Fargate Service with FireLens and Fluent Bit. I want to send logs to OpenSearch (or ElasticSearch). Two good options:

  1. Firelens / FluentBit -> kinesis plugin with compression and aggregation -> Kinesis Data Stream -> Kinesis Firehose with Lambda that decompresses and parses logs -> OpenSearch (or ElasticSearch).
  2. Firelens / FluentBit -> es plugin -> Open Search (or ElasticSearch).

First version saves bandwidth over the network and custom log parsing can be done.
But if no custom log parsing is needed, how can I measure that the second solution is better?
Is it better by the way?

1 Answer
0

While either option could work. Its typically better to air on the side caution and store the logs. You can always have the logs for later use with other services, audits, observability (across your entire stack). To that end, you can use the Kinesis solution for log preservation/parsing, but it also adds more moving parts to your solution. What i would do is

Firelens/Fluent Bit -> CloudWatch -> OpenSearch (or ES)

This gives you the best of both benefits of the solutions you highlighted. Unless there is another reason for Kenisis, i.e. real time analytics for example, this simplifies the pipeline considerably.

Here are some resources to get you started

Firelens/Fluent Bit -> CloudWatch https://aws.amazon.com/blogs/containers/fluent-bit-integration-in-cloudwatch-container-insights-for-eks/

CloudWatch -> OpenSearch https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html.

answered 9 months ago
  • I have CloudWatch parallel or next to the others like this:
    Firelens / FluentBit:

    • Cloudwatch plugin
    • kinesis plugin
    • es plugin
  • You only have to have the cloudwatch agent so no need for Kenisis (unless you need it for other reasons besides logging). Firelens/Fluent Bit willl rout and log and the CloudWatch agent will aggregate the logs for you in cloudwatch. Then you simply push the AWS Logs into OpenSearch. In the case I'm highlighting you will not need neither Kinesis plugin or ES plugin.

  • I appreciate your desire for simplicity. And for a Lambda, who has no FireLens next to it your solution will be the winning one. Just wanted to know a more scientific reason too. Nevertheless I will try to find some more advantage to it and post the answer here as soon as I find it.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions