Exploring AWS Solutions: Stream processing for 1000+ devices – Kinesis Data Streams, Firehose or Alternative?
0
I'm designing a system that will listen to over 1000 streams (1 for each device), and should be able to process the streamed data to take action when needed. I need the processing and taking action to be near-real-time, but not actually real time.
All devices streams start from a RPC call, so the system must:
Send the RPC call to attach it's session to the event stream.
Active listen to the stream, if certain events are received trigger a lambda (or make an API call, tbd)
Send keep-alive calls to maintain the session, every X seconds.
Active listen to the stream 24/7/365
No long term storage required.
I don't know if I can use Firehose for this.
Kinesis Data Stream seem to be a better match, however I don't know:
if it can make the RPC calls (start and keep alive), or if I would need another service in between.
If I would charged with 1000 streams per hour + data ingested? If so that would make it not financially feasible.