Data loss or duplication on Cloudtrail LookupEvents API StartTime parameter

0

I need to use LookupEvents API provied by Cloudtrial to periodically fetch events. Initially I set a **StartTime **parameter to something initially, and fetched all events. The after some time when I fetch, I specify the **StartTime **as recent event's **EventTime **from previous fetch + 1 second.

Can I use this? Or will I miss any events? If so, can I get any suggestions while also avoiding duplicate events.

1 Answer
3
Accepted Answer

Your approach of using the most recent event's EventTime + 1 second as the StartTime for the next lookup can generally work, but there is a small chance of missing events or having duplicate events if events have the same timestamp or if there is a delay in the processing and delivery of events.

To minimize the risk of missing or duplicating events, you can slightly modify your approach:

  1. Use the EndTime parameter along with StartTime. Set the EndTime to the current time when you make the API call. This ensures you're only fetching events within a specific time window.
  2. When setting the StartTime for the next API call, subtract a small buffer (e.g., 5 seconds) from the most recent event's EventTime. This will help account for events with the same timestamp and potential delays.

By implementing these changes, you will reduce the chance of missing or duplicating events. However, you may still need to handle duplicates on your side by storing and comparing event IDs or implementing deduplication logic.

profile picture
EXPERT
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions