Long running event processing


I am planning to build a long-running event processing system. I already have EC2 instances for my application and want to utilize the same compute to process long-running events. I am planning to use SQS with higher visibility timeouts and if the processing goes beyond that limit, I will change the message visibility time while processing. Can this pattern be used for scheduled events from the event bridge (do we have a concept of visibility timeout for event bridge events or do we have to send those to the SQS to get the same pattern)? Also, curious to know about ECS as a long-running compute for such events in the future (if I want to separate the compute from my existing ec2 compute), do we have to follow the same change in visibility timeouts for ECS as well if so?

1 Answer
Accepted Answer

I think you've answered all of your questions yourself (nice!) but to confirm:

Your thoughts around SQS and visibility timeouts are correct. What compute platform you use there (EC2 or ECS) is not crucial - it's the same logic working the same way.

EventBridge doesn't work the same way as SQS - there is no concept of event visibility in the sense that SQS has message visibility. But you could use EventBridge to schedule a message to be sent to SQS.

profile picture
answered a month ago
  • I also stumbled upon AWS Batch. I think I can use my existing compute with AWS Batch. Do you think AWS Batch would be a better fit both in terms of better dev experience (not using the SQS pattern with custom de-dup for effective one-time processing, etc) and cost?

  • Perhaps. You might also consider Step Functions.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions