CloudWatch log events not being returned in CLI and API

Hey, if you are using the GetLogEvents API call it is possible you may get empty responses.

From https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html

By default, this operation returns as many log events as can fit in a response size of 1MB (up to 10,000 log events). You can get additional log events by specifying one of the tokens in a subsequent call. This operation can return empty results while there are more log events available through the token.

So you can test the following Python code (you can use CloudShell in your AWS console which has boto3 setup). Just make sure to give the log group and the log stream name.

import boto3

#AWS_REGION = "us-east-1"
AWS_REGION = "ap-southeast-2"
log_client = boto3.client("logs", region_name=AWS_REGION)

params = {
    "logGroupName": "/aws-glue/crawlers",
    "logStreamName": "CustomCUR",
    "startFromHead": False,
}

log_events = []

while True:
    response = log_client.get_log_events(**params)
    #print(response)
    log_events.extend(response["events"])
    #print(log_events)
    next_token = response.get("nextBackwardToken")
    #print(next_token)
    if next_token == params.get("nextToken"):
        break
    params['nextToken'] = next_token
    # The following line is to validate the script is making the GLE calls and iterating through the next backward token
    print(next_token)

print(log_events)

The above code will run as long as the token returned is not same as the token passed. https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#API_GetLogEvents_ResponseSyntax

The token for the next set of items in the backward direction. The token expires after 24 hours. This token is not null. If you have reached the end of the stream, it returns the same token you passed in.

As mentioned above, this is a known issues that currently affect the GetLogEvents API (same applies to the FilterLogEvents API) whereby empty pages are returned despite events being present. The internal CloudWatch service team is aware of this and working on improvements. The issue can occur:

-- When there is a large log group with a log streams containing sparse events

-- When running GetLogEvents with no startTime/endTime (or a "wide" time window), CloudWatch returns little to no events.

-- Additionally, as stated in the API documentation below - " This operation can return empty results while there are more log events available through the token."

[+] https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html

Work Around:

-- ensure startTime/endTime are always set.

-- Using CloudWatch Log Insights instead:

[+] Analyzing log data with CloudWatch Logs Insights: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html

[+] Sample queries: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-examples.html

-- Use pagination when running either the GetLogEvents/FilterLogEvents. The only way to know for certain that the correct number of events has been returned is when no additional NextForwardToken is returned.

Thank you. I was able to resolve this by adding startFromHead = true to the request. It's unfortunate that these "known issues" are not documented anywhere. If I'm wrong about that, please send me a pointer.