Kinesis Analytics Application output to SNS?



I'm trying to figure out the best way to output messages to SNS from a Flink Kinesis Analytics application. My understanding is firehose has ~60s of latency, which is not acceptable for this use case, as we want <1 sec of latency from ingestion to output. If I'm ok with > once delivery of output, could I just create a flink function to send a message to SNS directly and not have to worry about creating a sink that handles checkpoints/deduplication/batching/whatever else?

We're looking at <1000 TPS.

Any thoughts or suggestions would be appreciated.


1 Answer

Thank you for reaching out to us. Firehose does have some latency due to buffering hints[1] in the service. Having Kinesis stream connector to kinesis analytics takes care of checkpoints/batching and this can be read by lambda and written to SNS. The architecture is as below:

Kinesis analytics --> Kinesis stream --> lambda --> SNS

Please refer to the below blog for example architecture : --

Please note that de-duplication should be done on consumer end. This can be done in lambda function in the above case. Let us know if you may have any other queries.


answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions