AWS Builder Center: Learn, Build and Connect with builders in the AWS community

AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.

Log events missing in cloudwatch export in S3 bucket using Create export task function

I am trying to create a daily backup of cloudwatch log to S3 bucket in log Archive account using lambda function calling cloud watch create export task. The export is happening successfully but I found there were few log event missing in log export log file for the given time range, though I can see log events in Cloudwatch stream in cloudwatch console.Also I can see the missing events TimeStamp and Ingestion time both are in the define time range. I tested both way, manual export using console menu and using Lambda. same result.

Can anyone suggest why this is happening. does anyone else also faced similar issue?

トピック: 管理とガバナンス
タグ: Amazon CloudWatch
言語: English

ANJU RANI

質問済み 1年前177ビュー

1回答

新しい順
投票が多い順
コメントが多い順

これらの回答は役に立ちましたか？コミュニティがあなたの知識を活用できるように、正解に賛成票を投じてください。

This could be becuase CloudWatch export tasks use the log ingestion time (when CloudWatch received the log), NOT the event timestamp shown in the log message. This is the most common reason for missing events. Export Range Uses ingestion time only. Common scenarios causing missing logs would be either Late Arriving Logs or Logs Still Being Ingested. One solution would be Add a Buffer Period and wait 5-10 minutes after the time period before running the export.

ref: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html

Manoj Kengudelu

回答済み 1年前

エキスパート

Matt Barbieri

レビュー済み 1年前

ANJU RANI
1年前
I understand, What I found is, though the ingestion time mentioned in CloudWatch is in my export task range, still log event missed in export. Additionally when i added buffer period of 10 minute, the the results are: example in my export log, one event is of time 23:58:01 and next event in log is from next date 00:05:02 but I can see almost 10 event in between these time range as missing if compared to Cloudwatch log events in Console. Not sure why this is happening.

Log events missing in cloudwatch export in S3 bucket using Create export task function

回答を追加する

関連するコンテンツ