encountering EventBridge - Kinesis Firehose requests limit threshold

Question

Hi there,  
i have an eventbridge rule configured with a kinesis firehose target to facilitate a bulk import effort. it seems to work well enough until i try and increase the throughput, wherein  i start to bump into firehose's 2000 requests/sec hard limit. judging by the firehose metrics, it appears as though EventBridge is submitting a single record per request ( IncomingRecords is the same as IncomingPutRequests )

my questions

1. is there a way to configure eventbridge to submit records to firehose in batches ( i am nowhere near exceeding the 500,000 records/sec nor the 5Mib/sec firehose limits ) ?
2. if 'no' on question 1, is there a recommended alternative arrangement ( still employing eventbridge and, ideally firehose, as i rely heavily on dynamic partitioning ) that may used?

Answer

Hi Craig.

You may be able to use [EventBrige Pipes](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html) for your use case. If you can apply filters to your ingested records, using pipes can help you distribute the records to different destinations (multiple Kinesis Firehose) which would help coping with the hard limits.

I hope this helps.

Answer

Besides EventBridge pipes mentioned in other answers, one other thing to consider is to send data from Event Bridge to Kinesis Data Stream first and from there to Kinesis Firehose. KDS has much higher rate limits and you can design your shards for the proper write bandwidth. At the integration point between KDS and KFH, it is KFH that will pull from KDS so theoretically you should not hit any rate limits.

Answer

Not sure if applicable, but have you considered using EventBridge pipes? You can either filter or use batching: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-batching-concurrency.html

encountering EventBridge -> Kinesis Firehose requests limit threshold

Conteúdo relevante