I'm designing a system that will listen to over 1000 streams (1 for each device), and should be able to process the streamed data to take action when needed. I need the processing and taking action to be near-real-time, but not actually real time.
All devices streams start from a RPC call, so the system must:
- Send the RPC call to attach it's session to the event stream.
- Active listen to the stream, if certain events are received trigger a lambda (or make an API call, tbd)
- Send
keep-alive
calls to maintain the session, every X seconds.
- Active listen to the stream 24/7/365
No long term storage required.
I don't know if I can use Firehose for this.
Kinesis Data Stream seem to be a better match, however I don't know:
- if it can make the RPC calls (start and keep alive), or if I would need another service in between.
- If I would charged with 1000 streams per hour + data ingested? If so that would make it not financially feasible.
Finally, I'm also open to other services ideas