Response from FilterLogEvents call, drops some events that exist in LogGroup

Issue:

When I call FilterLogEvents and repeat/loop the call, using nextToken from the response, to load all the results that match the query (same filter string, and time window), the aggregated result sometimes does not contain events that definitely exist in the LogGroup.

Context:

• Golang aws-sdk v1, FilterLogEvents call is looped, using nextToken, to load all events available
• start/end time, used in FilterLogEvents call, is dynamically set and can be many hours wide
• event rate can reach 100K per minute
• events come via single event bus (multiple event rules) into a single LogGroup (large number of log streams generated)
• issue usually seen at higher loads in LogGroup
• I am aware that above points may cause response with empty results but present nextToken and I account for that in my code, by making a the same call with nextToken

Debugging so far:

Missing events happen in 2 modes:

1. a single event (or a few events) is not returned in the response of ~500 events, whilst events with earlier and later timestamps are returned
2. after looping the call and applying nextToken in the next call, the following response returns empty events and nextToken=nil, whilst more events are definitely present with the later timestamps that fall within the start/end times specified in FilterLogEvents call.

I can see that events exist via aws console (using the same %filter query%) or by running the code again - however narrower time window was used in both cases. Ingestion time and timestamp of the "missing" events indicate that event should have been collected the first time round. Difference between Ingestion time and timestamp of the event is in order of 100ms, out of order evens only happen between events that are seconds apart. The time lag between "missing" event ingested and collected can be as large as 5-10 hours, yet the event can still be missing in the response, so "eventual consistency" effects are not to blame.