跳至内容

Log events missing in cloudwatch export in S3 bucket using Create export task function

0

I am trying to create a daily backup of cloudwatch log to S3 bucket in log Archive account using lambda function calling cloud watch create export task. The export is happening successfully but I found there were few log event missing in log export log file for the given time range, though I can see log events in Cloudwatch stream in cloudwatch console.Also I can see the missing events TimeStamp and Ingestion time both are in the define time range. I tested both way, manual export using console menu and using Lambda. same result.

Can anyone suggest why this is happening. does anyone else also faced similar issue?

已提问 1 年前173 查看次数

1 回答
0

This could be becuase CloudWatch export tasks use the log ingestion time (when CloudWatch received the log), NOT the event timestamp shown in the log message. This is the most common reason for missing events. Export Range Uses ingestion time only. Common scenarios causing missing logs would be either Late Arriving Logs or Logs Still Being Ingested. One solution would be Add a Buffer Period and wait 5-10 minutes after the time period before running the export.

ref: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html

已回答 1 年前

AWS
专家

已审核 1 年前

  • I understand, What I found is, though the ingestion time mentioned in CloudWatch is in my export task range, still log event missed in export. Additionally when i added buffer period of 10 minute, the the results are: example in my export log, one event is of time 23:58:01 and next event in log is from next date 00:05:02 but I can see almost 10 event in between these time range as missing if compared to Cloudwatch log events in Console. Not sure why this is happening.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。