Clarification Needed: GetIteratorAge Behavior in Kinesis Data Streams

Question

Description:
We're seeking clarification on the behavior of GetIteratorAge metric in Kinesis Data Streams (KDS), as we're observing what appears to be unexpected behavior with iterator aging.

Current Setup: Using GetRecords API to read from KDS
Observed Behavior:
1. When data ingestion stops:
    - GetIteratorAge continues to increase despite no new data
    - MillisBehindLatest shows high values even with no new records

Specific Example where with no records, MillisBehindLatest is high.
AWS CLI command output:
`aws kinesis get-records --shard-iterator XYZSSS --limit 100
{
    "Records": [],
    "NextShardIterator": "YYYYYYYYY",
    "MillisBehindLatest": 20470000
}`

Key Questions:
- Is it expected behavior for IteratorAgeMilliseconds to keep increasing when there are no new records and consumer keeps reading with the help of iterator?
- What is the relationship between MillisBehindLatest and GetIteratorAge metrics?

Documentation Reference:
As per AWS documentation: "The age of the last record in all GetRecords calls made against a Kinesis stream, measured over the specified time period."
We need clarification on:
**How GetIteratorAge is calculated when there are no new records**

re:Post Agent · Answer

The behavior you're observing with GetIteratorAge (also known as IteratorAgeMilliseconds) and MillisBehindLatest in Kinesis Data Streams is expected and can be explained by understanding how these metrics work.

1. GetIteratorAge (IteratorAgeMilliseconds) Behavior:
It is indeed expected for the IteratorAgeMilliseconds to keep increasing even when there are no new records being added to the stream. This metric represents the age of the most recent record received from all GetRecords calls to a data stream for a specified time window. When no new data is being ingested, this value will continue to increase because it's measuring the time since the last record was added to the stream.

2. MillisBehindLatest Behavior:
MillisBehindLatest indicates how far behind the current time the consumer is in processing records. It represents the difference between the current time and the timestamp of the latest record that could be returned in a GetRecords call. Even when there are no new records, this value can be high if the consumer hasn't caught up to the latest position in the stream.

3. Relationship between MillisBehindLatest and GetIteratorAge:
While these metrics are related, they measure slightly different things. GetIteratorAge focuses on the age of the last record processed, while MillisBehindLatest indicates how far behind the consumer is from the latest possible record. In scenarios where no new data is being added, both metrics can show high values, reflecting the time since the last record was added to the stream.

4. Calculation of GetIteratorAge with no new records:
When there are no new records, GetIteratorAge is still calculated based on the timestamp of the last record in the stream. It will continue to increase, reflecting the growing time gap between the current moment and when the last record was added.

It's important to note that these behaviors are by design and help in monitoring the overall health and performance of your Kinesis Data Streams application. High values in these metrics when there's no new data don't necessarily indicate a problem, but rather reflect the current state of the stream and consumer position.

To manage these metrics effectively, consider implementing logic in your application to handle scenarios where no new data is being ingested, such as adjusting polling frequency or implementing a backoff strategy when empty record sets are returned.  
**Sources**  
[Troubleshoot iterator age that continues to increase in Kinesis Data Streams | AWS re:Post](https://repost.aws/knowledge-center/msaf-iteratorage-metric)  
[Tutorial: Perform basic Kinesis Data Streams operations using the AWS CLI - Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/fundamental-stream.html)  
[GetRecords - Amazon Kinesis Data Streams Service](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html)  
[Troubleshoot Kinesis Data Streams issues in DynamoDB | AWS re:Post](https://repost.aws/knowledge-center/dynamodb-kinesis-data-streams-issues)

Clarification Needed: GetIteratorAge Behavior in Kinesis Data Streams

Relevant content