Why is my Lambda IteratorAge metric increasing, and how do I decrease it?
I see an increase (or spikes) in my AWS Lambda function's IteratorAge metric.
A Lambda function's iterator age increases when the function can't efficiently process the data that's written to the streams that invoke the function. To decrease your function's IteratorAge metric, you must increase your stream processing throughput.
The following factors influence a function's IteratorAge metric:
- Runtime duration
- Batch size
- Invocation errors
- Throttle occurrence
- Unevenly distributed records
- Shard count
Review this article to understand how each factor affects iterator age. Then, reconfigure your function or data stream to decrease your function's iterator age based on your use case. For more information about Lambda invocations, see Working with AWS Lambda function metrics.
Note: For Amazon DynamoDB streams, see Why is the Lambda IteratorAge metric increasing for my Amazon DynamoDB Streams?
Decrease your function's runtime duration
A high runtime duration increases a function's iterator age. Decreasing the duration increases a function's throughput, which decreases a function's iterator age.
To decrease your function's runtime duration, do one or both of the following:
1. Increase the amount of memory allocated to the function. Lambda allocates CPU power in proportion to the memory.
2. Optimize your function code so that less time is needed to process records.
Note: To get more information on your Lambda duration and performance, see Using AWS Lambda and AWS X-Ray.
Increase your stream's batch size
To increase your function's batch size, follow the instructions in Configuring a stream as an event source.
Note: If your function's duration is dependent on the number of records in an event, then increasing your function's batch size doesn't decrease the iterator age. For more information, see Working with streams.
Make sure that your function gracefully handles invocation errors
Invocation errors can cause a Lambda function to take longer to process an event, or process the same event more than once. Because event records are read sequentially, a function can't progress to later records if a record batch causes an error each time that it's retried. In this situation, the iterator age increases linearly as those records age.
If your function returns an error, Lambda continuously retries the batch. The batch retries continue until the processing is successful, the maximum retry attempts is reached, or the data expires from the stream.
Make sure that your function gracefully handles records written to the stream. As you develop your function, logging and instrumenting your code can help you diagnose errors.
For more information, see the following:
- Error handling and automatic retries in AWS Lambda
- Monitoring and troubleshooting Lambda applications
- New AWS Lambda controls for stream processing and asynchronous invocations
- Optimizing batch processing with custom checkpoints in AWS Lambda
Manage your throttle occurrence
Because event records are read sequentially, a function can't progress to the next record if the current invocation is throttled. In this situation, Iterator age will increase while Lambda retries the throttled invokes.
Evenly distribute records in the stream
Partition keys in each data record determine the shards in the stream that the records are written to. An increase in traffic to your stream with records containing the same partition key causes a shard to receive a disproportionate number of records. This results in a hot shard.
Kinesis enhanced shard-level metrics allow you to verify the ingestion rate into each shard and troubleshoot a hot shard.
For more information, see Under the hood: scaling your Kinesis data streams.
Increase your stream's shard count
A low number of shards in a stream increases a function's iterator age. Increasing the number of shards in a stream increases the number of concurrent Lambda functions consuming from your stream, which decreases a function's iterator age.
To increase your stream's shard count, follow the instructions in Resharding a stream.
Note: Shard splitting doesn't have an immediate effect on a function's iterator age. Existing records remain in the shards that they were written to. Those shards must catch up on their backlog before the iterator age for the shards decreases. For more information, see Working with streams.
Increase your stream processing throughput by testing different parallelization factor settings and using enhanced fan-out
To test parallelization factor settings
You can improve stream processing by configuring your function's parallelization factor to increase the number of concurrent Lambda invocations for each shard of a stream. This is done by specifying the number of concurrent batches that Lambda polls from each shard. This is configured on the event source configuration.
If your parallelization factor is set to 10, you can have up to 50 concurrent Lambda invocations to process five Kinesis data shards.
For example, concurrent runs = number of shards x concurrent batches per shard (parallelization Factor).
For more information, see Using AWS Lambda with Amazon Kinesis and New AWS Lambda scaling controls for Kinesis and Amazon DynamoDB event sources.
Note: When increasing the number of concurrent batches per shard, Lambda validates in-order processing at the partition key level.
To use enhanced fan-out
You can reduce latency and increase read throughput by creating a data stream consumer with enhanced fan-out. Stream consumers get a dedicated connection to each shard that doesn't impact other applications that are also reading from the stream.
For more information, see Developing custom consumers with dedicated throughput (enhanced fan-out) and Using AWS Lambda with Amazon Kinesis.
Note: It's a best practice to use enhanced fan-out if you have many applications reading the same data, or if you're reprocessing streams with large records.
- Uchiha_Madaralg...asked 2 years agolg...
- MWlg...asked 8 months agolg...
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated 5 months ago
- How to avoid network timeout issues when invoking long running Lambda functions from .NET6+ applications on Linux platformsEXPERTGabrielelg...published 6 months agolg...