DeliveryToS3.DataFreshness metrics exceed defined batch time + buffer

Here are a few points to clarify:

Data Freshness = Time in Buffer + Time in Retry Buffer + X seconds (Other processing time)

Time in Buffer: This is the time it takes for the data to move from the producer (source) to the Firehose delivery stream.

Time in Retry Buffer: This is the time it takes for the data to be retried if there were any delivery failures. Firehose retries to deliver the data for a certain amount of time before moving it to the retry buffer.

X seconds (Other processing time): This represents any additional processing time that might occur outside the buffer and retry buffer stages. This could include transformation or enrichment of the data.

Elaborating on how retries work in Amazon Kinesis Data Firehose:

When Firehose encounters a delivery failure (for example, due to service throttling or an internal error), it automatically retries to deliver the data for a specified period. During this retry period, Firehose attempts to redeliver the data to the destination (e.g., S3, Redshift, etc.) multiple times. If the data is successfully delivered within the retry period, it doesn't enter the retry buffer. If the data still cannot be delivered successfully after all the retry attempts, it is moved to the retry buffer. Data in the retry buffer is periodically retried according to the retry settings configured for your Firehose delivery stream.

Here is the link for RetryOptions: https://docs.aws.amazon.com/firehose/latest/APIReference/API_RetryOptions.html