Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

EventBridge duplicate events due to at-least-once delivery

0

EventBridge has at-least-once delivery. The event format is documented here: https://docs.aws.amazon.com/eventbridge/latest/ref/events-structure.html

However, what isn't documented is how exactly a possible duplicate event will look like. Does the $.id change for each duplicate? What about $.time? Or is the JSON payload absolutely identical (and thus a good candidate for SQS FIFO content-based deduplication)? Please let me know if you have any further info on this.

Another place where this comes up is when I trigger a State Machine from EventBridge Rule. StartExecution is idempotent for standard workflow if the name and input is the same. But how do I know the name/input will be the same for the duplicated events, if the event duplication is not properly documented ...

Topics
Application Integration
Tags
Amazon EventBridge
Language
English

asked a year ago1.4K views

2 Answers
5

Yes, AWS does not guarantee that duplicate events will be byte-for-byte identical, to handle the reliably:

  • Inject a UUID or deterministic hash into the event payload ($.detail) at the source.
  • Use that field as your deduplication key in downstream consumers (e.g., SQS FIFO MessageDeduplicationId or Step Functions executionName).
EXPERT

answered a year ago

  • Paya
    a year ago

    I do not have the ability to inject a UUID at the source for deduplication purposes, because these events are generated by AWS services, eg. EventBridge Rule triggered by a schedule/cron. How would I handle that?

1

It isn't just the EventBridge bus that can deliver the event more than once, but also the service that emits the event to EventBridge may do so repeatedly. Since the recipient generally has to be familiar with the context of events it receives and processes, it's generally best to rely on that contextual knowledge to choose a reliable way to deduplicate events.

For example, for events that notify about a resource or object being created, the resource, object, or object version often receives an ID that is guaranteed to be unique. For events about requests made, many APIs include a client token specifically meant for implementing idempotency that is also appropriate (possibly in combination with other request parameters) for deduplicating related notifications.

EXPERT

answered a year ago

EXPERT

reviewed a year ago

  • Riku_Kobayashi EXPERT
    a year ago

    Hello.

    The EventBridge Scheduler will publish the following event. The ID included in this event will be the same for duplicate events.

    https://docs.aws.amazon.com/eventbridge/latest/userguide/event-reference.html#event-detail-scheduled-event

    {
  "version": "0",
  "id": "89d1a02d-5ec7-412e-82f5-13505f849b41",
  "detail-type": "Scheduled Event",
  "source": "aws.events",
  "account": "123456789012",
  "time": "2016-12-30T18:44:49Z",
  "region": "us-east-1",
  "resources": ["arn:aws:events:us-east-1:123456789012:rule/SampleRule"],
  "detail": {}
}

    I think this can be solved by creating a mechanism to eliminate duplicates on the Lambda side that EventBridge calls, or by using StepFunctions and DynamoDB to evaluate the ID contained in the event before executing it.

  • Paya
    a year ago

    @Riku_Kobayashi: Where did you find that the ID included in the EventBridge Scheduler events will be the same for duplicate events? The other answer by Kidd Ip seems to suggest the opposite. I haven't been able to find any documentation on how duplicate events behave within the AWS Documentation, yet it is critical information to have in order to properly design code to work correctly when it encounters duplicated events.

  • Riku_Kobayashi EXPERT
    a year ago

    The ID included in the event is a UUID, so if the ID is the same, it can be determined to be a duplicate event. https://docs.aws.amazon.com/eventbridge/latest/ref/events-structure.html

    A Version 4 UUID that's generated for every event. You can use id to trace events as they move through rules to targets.

    In fact, you can confirm that an event is issued with the same ID in the question at the URL below. https://repost.aws/ja/questions/QUjRwgOSqdTIaDTtld3j3-YQ/duplicate-event-fired
    The EventBridge Scheduler allows you to pass additional context to a target, called <aws.scheduler.execution-id>, which is a unique ID assigned by the EventBridge Scheduler for each target invocation attempt. This can also be used to ensure idempotence. https://docs.aws.amazon.com/scheduler/latest/UserGuide/managing-schedule-context-attributes.html

  • Paya
    a year ago

    I have actually already seen the report.aws question you link to, but I concluded that it isn't a proof that every event will have the same ID. The EventBridge (Rule) is a distributed system, and depending on where the fault/lag might happen within the system, a duplicated event may be binary identical (perhaps more often than not), but without thorough examination of the implementation (or - having this be properly documented), it cannot be concluded that this will be the case for every single duplicated event.

    Thank you for the EventBridge Scheduler context attributes, it does look pretty promising. I do wish the user guide crystal clearly stated that duplicated events created due to at-least-once event delivery will have the same aws.scheduler.execution-id, so there is no room for speculation.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content