Why are my Kinesis Data Streams throttling?

Short description

Kinesis Data Streams throttles when write or read operations exceed per-shard throughput limits. However, your aggregated CloudWatch metrics might appear within quotas but throttling still occurs. This typically happens for the following reasons:

• Enhanced shard-level monitoring is turned off. This hides hot shards.
• Hidden micro spikes within the Amazon CloudWatch 60-second aggregation window.
• CloudWatch metrics only report successful operations. This masks the true request volume.

Resolution

Resolve ProvisionedThroughputExceededException or LimitExceededException errors

When you use the GetRecords or PutRecords, PutRecord API, your stream might produce ProvisionedThroughputExceededException or LimitExceededException throttling errors.

The errors can also occur for any of the following reasons.Write-side throttling

Write-side throttling causes the error when the following conditions are true:

• The number of records written per second exceeds 1,000 records/shard.
• The total data written exceeds 1 MiB/sec per shard.
• The total throughput in bytes exceeds the Kinesis stream limits.
• The producer makes too many rapid requests to write to the stream. A Slow down or Rate exceeded error usually shows this issue..

Read-side throttling

This error occurs when a single shard receives more than five GetRecords calls per second. The error can also happen when multiple consumers, without enhanced fan-out compete for the same shard's read capacity.

To diagnose this error, take the following actions:

• Check the WriteProvisionedThroughputExceeded metric for writes or ReadProvisionedThroughputExceeded metric for reads in CloudWatch. A non-zero value confirms throttling.
• Take the IncomingBytes or IncomingRecords metric value from a one-minute data point and divide by 60 to get the average per-second rate.
• If the average per-second rate doesn't exceed the quota, then add the IncomingRecords to the WriteProvisionedThroughputExceeded count. If the combined total exceeds the per-shard quota, then throttling is expected.
• Turn on logging on the data producer side to capture the total number and size of submitted records.

To resolve this issue, take the following actions:

• Increase the number of shards in your stream.
• Switch to On-Demand capacity mode for automatic scaling.
• For read-side throttling with multiple consumers, turn on enhanced fan-out to provide each consumer with dedicated 2 MiB/sec throughput per shard.

Enhanced shard-level monitoring is turned off

When enhanced monitoring is turned off, stream-level metrics are aggregated across all shards. This can hide hot shards, shards that receive a disproportionate amount of traffic because of uneven partition key distribution.

Example

A stream with 10 shards has a total write capacity of 10 MiB/sec. If your aggregated IncomingBytes shows 5 MiB/sec, then it appears healthy. However, if 80% of that traffic goes to a single shard at 4 MiB/sec, then that shard is throttling even though the stream-level metric looks fine.

The PutRecord****s or PutRecord.Success metric also shows operations that fail. When the success metric dips, review the data producer logs to find the root causes of the failures. If throttling occurs, then establish logging on the data producer side to determine the total number and size of submitted records. If the total number of records in the PutRecord or PutRecords calls exceed the stream quota, then your Kinesis stream throttles.

To diagnose this issue, take the following actions:

• Run the following command to turn on enhanced shard-level monitoring:

  aws kinesis enable-enhanced-monitoring \
    --stream-name <your-stream-name> \
    --shard-level-metrics ALL

  Note: Replace your-stream-name with your stream name
• Compare the IncomingBytes average and maximum values at the shard level. A large gap between them shows hot shards.
• Identify which shards have significantly higher IncomingBytes or IncomingRecords than others.

To resolve this issue, take the following actions:

• Use random or high-cardinality partition keys to distribute records evenly across shards.
• If you must use meaningful partition keys, then consider adding a random suffix to improve distribution.
• Monitor IncomingBytes and IncomingRecords at the shard level. When the Maximum and Average values are close, distribution is even.
• Consider re-sharding to split hot shards. Strategies for resharding.

Hidden micro spikes in CloudWatch metrics

Kinesis Data Streams quotas are enforced per second, but CloudWatch aggregates metrics over 60-second periods by default. A brief spike such as 3,000 records in one second might cause throttling even though the one-minute average of 50 records/sec is within limits.

Example

A shard receives 60,000 records in one minute. But if 5,000 of those records arrived in a single second, then that second experienced throttling. The CloudWatch metric won't reveal this.

To diagnose the issue, implement application-level logging to capture per-second record counts and sizes sent to each shard. Also check the WriteProvisionedThroughputExceeded metric. Even a small non-zero Sum value shows micro spikes occurred.

To resolve this issue, take the following actions:

• Implement exponential backoff with jitter in your producer retry logic.
• Use the Kinesis Producer Library (KPL) which provides built-in rate limiting and record aggregation.
• Smooth out traffic by buffering records and sending them at a consistent rate.
• Increase the number of shards to provide headroom for bursts.
• Limit the request rate of the producer and the number of records that are sent per second to match the capacity of the stream.

CloudWatch metrics only report successful operations

CloudWatch Kinesis metrics (IncomingRecords, IncomingBytes) only count records that Kinesis Data Streams successfully ingests. Throttled or failed records are excluded from these metrics. This means the actual volume of traffic hitting your stream is higher than what CloudWatch shows.

When failed records can't enter the Kinesis stream, the stream throttles. If a retry behavior is in the producer, then the producer retries failed records. The process might then get delayed.

To check whether too many records are being sent to the Kinesis stream, add the total number of incoming records to the number of throttled records.

Use the following formula to calculate the request volume:

True volume = IncomingRecords (Sum) + WriteProvisionedThroughputExceeded (Sum)

If the true volume exceeds your stream's capacity, then throttling is expected. Even when IncomingRecords alone appears within limits.

Review producer-side logs for retry counts. High retry rates show that failed records are resubmitted and further increase load.

To resolve the issue, take the following actions:

• Scale your stream to accommodate the true request volume, not just the successful volume. Add shards or switch to On-Demand mode.
• Implement circuit breaker patterns in your producer to avoid retry storms.
• Use the KPL's built-in rate limiting to prevent overwhelming the stream.

Related information

Developing custom consumers with dedicated throughput (enhanced fan-out)

Monitoring the Amazon Kinesis Data Streams Service with Amazon CloudWatch

Enable-enhanced-monitoring

Kinesis Data Streams quotas and limits