EventBridge or SQS immediate for first item then batching

Question

A client has asked us to create an API endpoint that they can hit with changes to a system. They have asked that these changes happen ASAP, but also the changes might be a one off or they might send a large batch of changes - but these will be individual API calls.

There are multiple systems, and the changes they send will have an ID that determines which system they want to affect. Each system can have multiple entries. The below is VERY simplified but Value would be a large JSON blob containing unique information.

This will be stored in DynamoDB something like the following;

UID	SystemID	Value
1	1		2
2	2		3
3	2		5
4	1 		10
5	2		8

The ideal situation is that we have an Event Bridge trigger on the DynamoDB which hits a Lambda which then pushes the changes to the individual system, this is where it gets tricky as the "system" can only accept a new payload with ALL of the data it needs.

For one of changes that works great, but if the client sends multiple changes through then what we'd really like to do is do the first one immediately (incase that is the only one) and then any subsequent ones wait 5 minutes, I would say at this point do them in a batch, but in reality we only need to do a single one as it's essentially a DynamoDB query to JSON, so all changes would be captured.

I've been looking at SQS queues, Step Machines and a few other options, but cannot work out how I would fathom this to work. If anybody has any suggestions that would be amazing.

re:Post Agent · Answer

Based on your requirements, you need a solution that processes the first change immediately while batching subsequent changes. Here's how you could approach this:

You can use a combination of DynamoDB Streams and SQS to achieve your goal:

1. Set up DynamoDB Streams on your table to capture item-level modifications in real-time.

2. Create a Lambda function triggered by the DynamoDB Stream that:
- Identifies which system the change belongs to (using SystemID)
- For each SystemID, checks if this is the first change in a time window
- If it's the first change, processes it immediately
- If it's a subsequent change, sends a message to an SQS queue with a delay

3. Configure an SQS queue with a message delay of 5 minutes for batching purposes.

4. Create another Lambda function triggered by the SQS queue that:
- When triggered, queries the entire dataset for the specific SystemID
- Transforms the data into the required format
- Sends the complete dataset to the target system

This approach gives you the immediate processing for the first change while batching subsequent changes. SQS is better suited for this batching requirement than EventBridge, as EventBridge doesn't have built-in batching capabilities.

For audit and compliance purposes, you could also log all successful transactions and failed records to maintain a comprehensive audit trail of the data processing.

If you need to scale this solution for larger datasets, you can adjust the Lambda memory, timeout settings, and increase ephemeral storage as needed.  
**Sources**  
[can AWS Event Bridge replace SQS? | AWS re:Post](https://repost.aws/questions/QUQ93OQzLQRVGx_ViiXpxqLg/can-aws-event-bridge-replace-sqs)  
[Ingest CSV data to Amazon DynamoDB using AWS Lambda | AWS Database Blog](https://aws.amazon.com/blogs/database/ingest-csv-data-to-amazon-dynamodb-using-aws-lambda/)  
[Write to Kinesis Data Streams using Amazon DynamoDB - Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/using-other-services-ddb.html)

EventBridge or SQS immediate for first item then batching

Add your answer

Relevant content