Bug on CloudWatch filters on API Javascript v3

The issue you're experiencing with CloudWatch filters in the AWS SDK for JavaScript v3 is likely related to how the GetLogEventsCommand handles pagination and the endTime parameter.

When you use the nextToken for subsequent requests, the API ignores the startTime and endTime parameters. This is by design to ensure consistency in paginated results. The nextToken essentially bookmarks where the previous request left off, regardless of the time range originally specified.

To address this and ensure you only get logs within your desired time range, you'll need to implement additional logic in your code:

1. Continue using startTime and endTime in your initial request.
2. For subsequent requests using nextToken, you'll need to manually filter the results to discard any events outside your desired time range.
3. Keep track of the last event timestamp you've processed and use it to determine when to stop paginating.

Here's a modified version of your fetchLogs function that implements this approach:

async function fetchLogs(logGroupName, logStreamName, startDate, endDate, nextPage = null) {
    const parameters = {
        logGroupName: logGroupName,
        logStreamName: logStreamName,
        startFromHead: false,
        unmask: false,
    };

    if (nextPage === null) {
        parameters.startTime = startDate;
        parameters.endTime = endDate;
    } else {
        parameters.nextToken = nextPage;
    }

    const command = new GetLogEventsCommand(parameters);
    const response = await cloudWatchClient.send(command);

    // Filter events to ensure they're within the desired time range
    response.events = response.events.filter(event => 
        event.timestamp >= startDate && event.timestamp <= endDate
    );

    // Check if we've reached the end of our desired time range
    const lastEventTimestamp = response.events[response.events.length - 1]?.timestamp;
    if (lastEventTimestamp && lastEventTimestamp > endDate) {
        response.nextForwardToken = null; // Stop pagination
    }

    return response;
}

This modified function will filter the events to ensure they fall within your specified time range, even when using nextToken for pagination. It also checks if the last event in the current batch exceeds your endDate, in which case it sets the nextForwardToken to null to stop further pagination.

Remember to handle the case where the response might be empty after filtering in your main logic that calls this function.

This approach should resolve the issue of getting records outside your specified date range while still allowing you to paginate through large sets of log events efficiently.

Sources
CloudWatch examples using SDK for JavaScript (v3) - AWS SDK for JavaScript
CloudWatch Logs examples using SDK for JavaScript (v3) - AWS SDK for JavaScript
CloudWatch Logs examples using SDK for JavaScript (v3) - AWS SDK Code Examples

The main problem is the parameter startFromHead ! It seems the startFromHead false parameter do a reverse order. The documentation does not show if it is false, the backwardToken attribute should be used on response. I was using nextToken and startFromHead false. If one use nextToken it still works !!! AWS returns records out of bound filter... Very weird behaviour on AWS API, but I guess I found the problem and a solution.

The problem reported is kept. Even though the code handle manually if the range is out of dates, the CloudWatchAPI returns records outside the range and it also does not returns every records inside the range.

async function fetchLogs(logGroupName, logStreamName, startTime, endTime, nextPage = null) {
    const parameters = {
        logStreamName: logStreamName,
        logGroupName: logGroupName
    };

    if (nextPage !== null) {
        parameters.nextToken = nextPage;
    } else {
        parameters.startFromHead = true;
        parameters.unmask = false;
        parameters.startTime = startTime;
        parameters.endTime = endTime;
    }

    // https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/cloudwatch-logs/command/GetLogEventsCommand/
    const command = new GetLogEventsCommand(parameters);

    const result = await cloudWatchClient.send(command);
    result.end = false;
    result.logs = [];

    if (result.$metadata?.httpStatusCode === 200) {
        for (let i = 0; i < result.events.length; i++) {
            const e = result.events[i];
         
            if (e.timestamp > dataFim) {
                result.end = true;
                console.log(`Timestamp ${e.timestamp} greater than EndTIme ${endTime}`);
                break;
            }
            result.logs.push(result.events[i]);
        }
        delete result.events;
    }
    return result;
}