How do I resolve throttle on my Firehose delivery stream when incoming traffic is within the quota?

Resolution

If a Firehose delivery stream's data ingestion rate exceeds the throughput capacity quota, then throttle occurs. If your stream's data ingestion rate looks to be within the Firehose quota but you still experience throttle, then your stream might have hidden microbursts.

Microburst throttle occurs because of sudden, high-volume data spikes that exceed your stream's throughput capacity for a short time. For example, if you use the PutRecord or PutRecordBatch API operation, then your data producers might send data in bursts that exceed your stream's per-second throughput quota.

Firehose publishes Amazon CloudWatch metrics every minute, however Firehose defines throughput capacity quotas by the second. So, if bursts of incoming data occur only for a few seconds, then the IncomingRecords, IncomingBytes, and IncomingPutRequest CloudWatch metrics don't capture the incoming data burst. However, the ThrottledRecords metric shows as Throttling.

To resolve microburst throttle on your Firehose delivery stream, take one or more of the following actions:

• Use an exponential backoff and retry behavior in the producer logic.
• Check the sizes and number of records that your producer sends. If the combined total of the incoming and throttled records is greater than the stream quota, then change the size or number of records.
• Create CloudWatch alarms for the IncomingBytes, IncomingRecords, IncomingPutRequests, and ThrottledRecords metrics.
• Contact AWS Support to request a throughput quota increase.
  Note: Firehose automatically increases the throughput quota of Direct PUT streams that experience throttle because of increased data ingestion until the throttle stops.