Skip to content

CloudWatch writing to expired logstreams?

0

We have a log group (with a 1-day expiry) set to receive events from Route53. Of course, these get split into multiple streams. We have an app that queries the last few hours of logs by requesting all the streams associated with the group and then filtering based on the last ingestion time before performing a regex on the remaining text.

For the most part, this works well but we have been known to miss messages. While looking into this, I found a recently written message (05/09/23 13:59:00) in a logstream that was marked as expired with a last ingestion date/time of 05/07/23 02:17:58.

By 15:37, as I am writing this, the last ingestion date on the expired logstream had finally updated. I understand these fields are eventually consistent, but I always thought of this as being a matter of seconds, not over an hour.

Given that this kind of delay isn't acceptable to our application, is there a better way to filter these messages or even receive them directly from Route53? Even if I took off the time-window in my call to DescribeLogStreamsRequest() I'd need to widen out further to get those marked as "expired" to cover this scenario, so I'm at a loss as to how this can be improved.

1 Answer
0

Hello,

Greetings from AWS Premium Support !

From the case notes, I undertsand that you have observed delay in updation of last ingestion date on the expired logstream.

Please correct me If I misunderstood your query.

-> As you said, While looking into this, I found a recently written message (05/09/23 13:59:00) in a logstream that was marked as expired with a last ingestion date/time of 05/07/23 02:17:58. So, Can you please elaborate about expired log streams?, means Is it related to deleted log stream?, please confirm.

-> As you are observing huge delay, So are you facing this issue with particular log group or other log group also.

To ivestigate this type of issue, We will have to check on the log group level to find root cause. So I would like to request you to Please reachout to our Support team by opening a case with same account in which resources are available, and they would be able to check in detailed way.

I will await your response.

Have a great day ahead.

AWS
SUPPORT ENGINEER
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.