Exploring AWS Solutions: Stream processing for 1000+ devices – Kinesis Data Streams, Firehose or Alternative?

Question

I'm designing a system that will listen to over 1000 streams (1 for each device), and should be able to process the streamed data to take action when needed. I need the processing and taking action to be **near-real-time**, but not actually real time.

All devices streams start from a RPC call, so the system must:
1. Send the RPC call to attach it's session to the event stream.
2. Active listen to the stream, if certain events are received trigger a lambda (or make an API call, tbd)
3. Send `keep-alive` calls to maintain the session, every X seconds.
4. Active listen to the stream 24/7/365
*No long term storage required.*

I don't know if I can use Firehose for this. 
Kinesis Data Stream seem to be a better match, however I don't know:
1. if it can make the RPC calls (start and keep alive), or if I would need another service in between. 
2. If I would charged with 1000 streams per hour + data ingested? If so that would make it **not** financially feasible.

Finally, I'm also open to other services ideas

Giovanni Lauria · Answer

You can use Amazon Kinesis Data Streams for ingesting and processing high-velocity data streams.

However, Kinesis Data Streams doesn’t directly handle RPC calls or session management. Consider using AWS Lambda to process incoming data from Kinesis.

For RPC calls and session management, you’ll need an additional service (e.g., an API Gateway).

Keep in mind the cost implications of using Kinesis Data Streams with 1000 streams.

Exploring AWS Solutions: Stream processing for 1000+ devices – Kinesis Data Streams, Firehose or Alternative?

Relevanter Inhalt