Cloudwatch Logs Insights scans the whole LogGroup, even when filtered to a specific LogStream

0

I am using Cloudwatch Logs Insights in this manner:

     fields @timestamp, @message, @logStream, @log
     | filter @logStream='xxxxxxxx'
     | filter field1="yyyyyyy"
     | sort @timestamp desc
     | limit 20

When I run this query, I expect only the logStream='xxxxxxxx' to be scanned. However, CloudWatch Insights scans the entire logGroup. This makes every query very very expensive.

Is there a way to limit the data scanned to only a specific logStream?

vmit
質問済み 4ヶ月前390ビュー
2回答
1

Based on your question, you are trying to restrict the way the Logs Insights query data and you want to query only specific log stream. CloudWatch Logs Insights primarily allows you to query and analyze data at Log Group level. In another words, it operates on log groups rather than individual log streams.

If you want to analyze data from specific log stream, I highly suggest you to export your logs (and log streams) to Amazon S3 and query data using Athena.

See this: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html#S3PermissionsConsole and https://aws.amazon.com/blogs/aws/amazon-athena-interactive-sql-queries-for-data-in-amazon-s3/

If you find this useful, please accepted the answer.

Regards

AWS
Takeda
回答済み 4ヶ月前
1

At present, the factors that govern CloudWatch Logs Insights scan size are:

  1. Log Group selection
  2. Time range selection

Filtering is done post-scan, so the filter by log group criteria does not limit the scanned volume. Your options for reducing Logs Insights scan volume within CloudWatch are to either 1/ limit the time range selection 2/ consider mapping data within your Log Group to multiple Log Groups.

Matt T
回答済み 4ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン