By using AWS re:Post, you agree to the Terms of Use

Kinesis Analytics Application output to SNS?



I'm trying to figure out the best way to output messages to SNS from a Flink Kinesis Analytics application. My understanding is firehose has ~60s of latency, which is not acceptable for this use case, as we want <1 sec of latency from ingestion to output. If I'm ok with > once delivery of output, could I just create a flink function to send a message to SNS directly and not have to worry about creating a sink that handles checkpoints/deduplication/batching/whatever else?

We're looking at <1000 TPS.

Any thoughts or suggestions would be appreciated.


1 Answers

Thank you for reaching out to us. Firehose does have some latency due to buffering hints[1] in the service. Having Kinesis stream connector to kinesis analytics takes care of checkpoints/batching and this can be read by lambda and written to SNS. The architecture is as below:

Kinesis analytics --> Kinesis stream --> lambda --> SNS

Please refer to the below blog for example architecture : --

Please note that de-duplication should be done on consumer end. This can be done in lambda function in the above case. Let us know if you may have any other queries.


answered 17 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions