can AWS Event Bridge replace SQS?

Hello,

EventBridge cannot replace SQS if your main requirement is batching. However, you could potentially use EventBridge + a Lambda, which would be triggered more frequently than needed but would execute the desired processing only when your threshold is met. This could be achieved by querying your S3 bucket via the SDK, using a database or ElastiCache if you need to store intermediate data, etc.

Hope it helps :)

While Amazon EventBridge can replace some functionalities of Amazon SQS, it may not be the ideal replacement for your specific use case of batching files before triggering a Lambda function.

EventBridge is primarily designed for event-driven architectures and real-time event processing. It excels at routing events from various sources to multiple targets based on rules. However, it doesn't have built-in batching capabilities like SQS.

For your specific requirement of triggering a Lambda function only when you achieve a batch of 30 files, SQS is better suited. SQS allows you to configure batch sizes for Lambda triggers, which is exactly what you need. You can set up your SQS queue to accumulate messages (each representing a file upload) and only invoke your Lambda function when the batch size reaches 30.

If you want to avoid triggering Lambda for each file upload to S3, you can still use EventBridge in combination with SQS:

1. Set up an EventBridge rule to capture S3 file upload events.
2. Configure the rule to send these events to an SQS queue instead of directly to Lambda.
3. Set up your SQS queue with a batch size of 30 for the Lambda trigger.

This approach would allow you to maintain the batching functionality you need while still leveraging EventBridge for event routing. However, replacing SQS entirely with EventBridge for your batching requirement would be challenging and likely less efficient.

In summary, while EventBridge is a powerful service for event-driven architectures, it's not designed to replace SQS's batching capabilities. For your specific use case, continuing to use SQS for batching, possibly in combination with EventBridge for event routing, would be the recommended approach.
Sources
Amazon EventBridge Features | Event Bus | Amazon Web Services
Automating notifications from AWS services to Amazon SQS using Amazon EventBridge - Amazon Simple Queue Service

For your use case EventBridge is not a good option. You can send the events from S3 to SQS and use a batch size of 30 with a large batch window so that the function triggers when there are 30 new files.

If you need to use EventBridge, you will need to invoke the function for every file. You can count the files in the function (and save the number of files in a DDB table for example) and when you get enough files, you actually do the processing.