Why does the Kinesis Firehose output "Throttling error encountered when calling Kinesis".

0

I use a Kinesis Data Stream connected to a Kinesis Firehose (Delivery stream). Some data failed to write a below message.

Throttling error encountered when calling Kinesis. This can be due to other applications calling the same APIs as the Firehose delivery stream, or because you have created too many Firehose delivery streams with the same Kinesis stream as the source.

I never used ether multiple data streams or multiple firehoses. Why did this message output?

질문됨 2년 전4742회 조회
3개 답변
1

How much data are you trying to move through the data stream to the firehose to S3 (per second)?

Kinesis Data Stream size, write, etc limits (and how to increase shards) is linked > here < Hopefully it's something straightforward like capacity, and can be resolved with an extra shard.

Troubleshooting on the Firehose service is > here < I'd recommend looking at the data freshness metric section - that talks about throttling.

AWS
Dan_H
답변함 2년 전
0

Thank you for your answer. Your answer seems to be correct.
My Kinesis Data stream reported the throttling issue on reads. And this warning was not important because Kinesis Firehose could retry. After a short wait, the data was correctly recieved.

답변함 2년 전
0

Hi there, the issue occured might due to you have multiple consumers for the Kinesis data stream, for the limitation, the Kinesis Data Fireshose use GetRecords API to retrieve data from data stream, for GetRecords API, it has 5 TPS for hard limit, which means you cannot invoke the API 5 times per second.

If the issue persist, you could try to do the following methods to optimize:

  1. recude the consumer count
  2. try to slow down the polling rate for other consumers (since you cannot control the pulling rate for Firehose)
  3. enable Enhanced fanout for your other consumers
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠