Kinesis Analytics Application output to SNS?

Question

Hello,

I'm trying to figure out the best way to output messages to SNS from a Flink Kinesis Analytics application. My understanding is firehose has ~60s of latency, which is not acceptable for this use case, as we want <1 sec of latency from ingestion to output. If I'm ok with > once delivery of output, could I just create a flink function to send a message to SNS directly and not have to worry about creating a sink that handles checkpoints/deduplication/batching/whatever else?

We're looking at <1000 TPS.

Any thoughts or suggestions would be appreciated.

Thanks!

Answer

Thank you for reaching out to us. Firehose does have some latency due to buffering hints[1] in the service. Having Kinesis stream connector to kinesis analytics takes care of checkpoints/batching and this can be read by lambda and written to SNS. The architecture is as below:

Kinesis  analytics --> Kinesis stream --> lambda --> SNS

Please refer to the below blog for example architecture :
-- https://aws.amazon.com/blogs/big-data/building-a-real-time-notification-system-with-amazon-kinesis-data-streams-for-amazon-dynamodb-and-amazon-kinesis-data-analytics-for-apache-flink/

Please note that de-duplication should be done on consumer end. This can be done in lambda function in the above case. Let us know if you may have any other queries.

References
~~~~~~~~~
[1] https://docs.aws.amazon.com/firehose/latest/APIReference/API_BufferingHints.html

Kinesis Analytics Application output to SNS?

Relevant content