Kinesis Analytics Application output to SNS?



I'm trying to figure out the best way to output messages to SNS from a Flink Kinesis Analytics application. My understanding is firehose has ~60s of latency, which is not acceptable for this use case, as we want <1 sec of latency from ingestion to output. If I'm ok with > once delivery of output, could I just create a flink function to send a message to SNS directly and not have to worry about creating a sink that handles checkpoints/deduplication/batching/whatever else?

We're looking at <1000 TPS.

Any thoughts or suggestions would be appreciated.


1개 답변

Thank you for reaching out to us. Firehose does have some latency due to buffering hints[1] in the service. Having Kinesis stream connector to kinesis analytics takes care of checkpoints/batching and this can be read by lambda and written to SNS. The architecture is as below:

Kinesis analytics --> Kinesis stream --> lambda --> SNS

Please refer to the below blog for example architecture : --

Please note that de-duplication should be done on consumer end. This can be done in lambda function in the above case. Let us know if you may have any other queries.


지원 엔지니어
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠