Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Log events missing in cloudwatch export in S3 bucket using Create export task function

0

I am trying to create a daily backup of cloudwatch log to S3 bucket in log Archive account using lambda function calling cloud watch create export task. The export is happening successfully but I found there were few log event missing in log export log file for the given time range, though I can see log events in Cloudwatch stream in cloudwatch console.Also I can see the missing events TimeStamp and Ingestion time both are in the define time range. I tested both way, manual export using console menu and using Lambda. same result.

Can anyone suggest why this is happening. does anyone else also faced similar issue?

Topics
Management & Governance
Tags
Amazon CloudWatch
Language
English

asked 10 months ago169 views

1 Answer
0

This could be becuase CloudWatch export tasks use the log ingestion time (when CloudWatch received the log), NOT the event timestamp shown in the log message. This is the most common reason for missing events. Export Range Uses ingestion time only. Common scenarios causing missing logs would be either Late Arriving Logs or Logs Still Being Ingested. One solution would be Add a Buffer Period and wait 5-10 minutes after the time period before running the export.

ref: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html

answered 10 months ago

AWS
EXPERT

reviewed 10 months ago

  • ANJU RANI
    10 months ago

    I understand, What I found is, though the ingestion time mentioned in CloudWatch is in my export task range, still log event missed in export. Additionally when i added buffer period of 10 minute, the the results are: example in my export log, one event is of time 23:58:01 and next event in log is from next date 00:05:02 but I can see almost 10 event in between these time range as missing if compared to Cloudwatch log events in Console. Not sure why this is happening.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content