Long running event processing

Question

I am planning to build a long-running event processing system. I already have EC2 instances for my application and want to utilize the same compute to process long-running events. I am planning to use SQS with higher visibility timeouts and if the processing goes beyond that limit, I will change the message visibility time while processing. Can this pattern be used for scheduled events from the event bridge (do we have a concept of visibility timeout for event bridge events or do we have to send those to the SQS to get the same pattern)? Also, curious to know about ECS as a long-running compute for such events in the future (if I want to separate the compute from my existing ec2 compute), do we have to follow the same change in visibility timeouts for ECS as well if so?

Accepted Answer

I think you've answered all of your questions yourself (nice!) but to confirm:

Your thoughts around SQS and visibility timeouts are correct. What compute platform you use there (EC2 or ECS) is not crucial - it's the same logic working the same way.

EventBridge doesn't work the same way as SQS - there is no concept of event visibility in the sense that SQS has message visibility. But you could use EventBridge to schedule a message to be sent to SQS.

Long running event processing

관련 콘텐츠