Kinesis Analytics Application output to SNS?



I'm trying to figure out the best way to output messages to SNS from a Flink Kinesis Analytics application. My understanding is firehose has ~60s of latency, which is not acceptable for this use case, as we want <1 sec of latency from ingestion to output. If I'm ok with > once delivery of output, could I just create a flink function to send a message to SNS directly and not have to worry about creating a sink that handles checkpoints/deduplication/batching/whatever else?

We're looking at <1000 TPS.

Any thoughts or suggestions would be appreciated.


1 Antwort

Thank you for reaching out to us. Firehose does have some latency due to buffering hints[1] in the service. Having Kinesis stream connector to kinesis analytics takes care of checkpoints/batching and this can be read by lambda and written to SNS. The architecture is as below:

Kinesis analytics --> Kinesis stream --> lambda --> SNS

Please refer to the below blog for example architecture : --

Please note that de-duplication should be done on consumer end. This can be done in lambda function in the above case. Let us know if you may have any other queries.


beantwortet vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen