By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Best way to have failsafe using kinesis and firehose?

0

Hey everyone,

I'm currently working with a lot of data consumption via kinesis stream and firehose, previously i used only firehose with direct puts but it seems that if you have a lot of throughput using kinesis streams is the better option. After changing to kinesis stream i noticed everything improved and i finally felt i was not being bottlenecked, thing is i want to have an architecture that is future proof and scales and i wanted to know if anyone with experience handling a lot of data with this setup has any opinion on this.

Basically from what i've read and from the data usage itself, it seems that if you use kinesis in on demand mode, basically you have almost infinite throughput (i think you can get up to 2GB/second), but i'm not sure if that applies to individual shards or if it applies to the whole stream. If it applies shard-wise then the only thing that can possibly bottleneck me would be firehose right?

So i wanted to know if my assumptions are correct bc the documentation is really confusing and does not details this part really well, and if they are not what are the real limitations of these two services so i can create something really reliable

Topics
Architecture StrategyAnalyticsInternet of Things (IoT)
Tags
Architecture StrategyAmazon Data FirehoseAmazon KinesisService Quotas
Language
English
rePost-User-5897466
asked 4 months ago430 views
1 Answer
0

Please refer the blog https://aws.amazon.com/blogs/big-data/amazon-kinesis-data-streams-on-demand-capacity-mode-now-scales-up-to-1-gb-second-ingest-capacity/ which talks about Kinesis Data Streasm scenarios. For majority of the cases on-demand mode is the best but during major event like when you anticipate a sudden surge , you can start with provisioned mode and then switch to on-demand. This will help you set the water mark at provisioned capacity limits even after switching to on-demand mode. Also keep in mind provisioned mode is costly so its better to use it for certain peak load cases only. To answer your questions, the throughput applies at stream level. If you need higher limits, you can request a limit increase through AWS Support with valid use case justification

AWS
rePost-VijayKr
answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content