encountering EventBridge -> Kinesis Firehose requests limit threshold

1

Hi there,
i have an eventbridge rule configured with a kinesis firehose target to facilitate a bulk import effort. it seems to work well enough until i try and increase the throughput, wherein i start to bump into firehose's 2000 requests/sec hard limit. judging by the firehose metrics, it appears as though EventBridge is submitting a single record per request ( IncomingRecords is the same as IncomingPutRequests )

my questions

  1. is there a way to configure eventbridge to submit records to firehose in batches ( i am nowhere near exceeding the 500,000 records/sec nor the 5Mib/sec firehose limits ) ?
  2. if 'no' on question 1, is there a recommended alternative arrangement ( still employing eventbridge and, ideally firehose, as i rely heavily on dynamic partitioning ) that may used?
profile picture
craig
asked a year ago405 views
3 Answers
0

Not sure if applicable, but have you considered using EventBridge pipes? You can either filter or use batching: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-batching-concurrency.html

profile picture
EXPERT
answered a year ago
0

Hi Craig.

You may be able to use EventBrige Pipes for your use case. If you can apply filters to your ingested records, using pipes can help you distribute the records to different destinations (multiple Kinesis Firehose) which would help coping with the hard limits.

I hope this helps.

profile pictureAWS
EXPERT
answered a year ago
0

Besides EventBridge pipes mentioned in other answers, one other thing to consider is to send data from Event Bridge to Kinesis Data Stream first and from there to Kinesis Firehose. KDS has much higher rate limits and you can design your shards for the proper write bandwidth. At the integration point between KDS and KFH, it is KFH that will pull from KDS so theoretically you should not hit any rate limits.

AWS
EXPERT
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions