firehose retry behaviors

Question

I heard some responses from AWS support about firehose retry, but want to address my confusion

### Question got answered before

> Will record saved in S3 backup bucket auto-reprocess? If so, how many times firehose would retry? How long will firehose keeping retry until succeed?

Ans.  Unfortunately, the records that are backed up to the S3 bucket due to delivery failures will not automatically reprocessed by Firehose. However, please note that if data delivery to your Amazon S3 bucket fails, Amazon Kinesis Data Firehose retries to deliver data every 5 seconds for up to a maximum period of 24 hours. If the issue continues beyond the 24-hour maximum retention period, it discards the data.

[+] Data Delivery Failure Handling - https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#retry  
 
 If Firehose encounters errors while delivering or processing data, it retries until the configured retry duration expires. If the retry duration ends before the data is delivered successfully, Firehose backs up the data to the configured S3 backup bucket. If the destination is Amazon S3 and delivery fails or if delivery to the backup S3 bucket fails, Firehose keeps retrying until the retention period ends.

For DirectPut delivery streams, Firehose retains the records for 24 hours. For a delivery stream whose data source is a Kinesis data stream, you can change the retention period as described in Changing the Data Retention Period.

[+] https://docs.aws.amazon.com/streams/latest/dev/kinesis-extended-retention.html

### Confusion
Seems there's no additional configuration for retry from below paragraph. And the bahavior of firehose to retry per 5 sec for 24 hours is a default behavior: 
> Amazon Kinesis Data Firehose retries to deliver data every 5 seconds for up to a maximum period of 24 hours. If the issue continues beyond the 24-hour maximum retention period, it discards the data.

But in the second paragraph, it mentioned configured retry duration. Is that refering retry duration in Dynamic partitioning. Does that mean if I don't enable Dynamic partitioning, Firehose would not retry at all?

> If Firehose encounters errors while delivering or processing data, it retries until the configured retry duration expires. If the retry duration ends before the data is delivered successfully

Could you elaborate on this please?

Giovanni Lauria · Answer

The “configured retry duration” in Kinesis Data Firehose refers to the total time spent on retries after the initial attempt to send data to the destination fails.
This retry behavior is a default feature of Kinesis Data Firehose and does not depend on whether Dynamic Partitioning is enabled or not.
The default value for the retry duration is 3600 seconds (60 minutes) for Redshift and can be up to 7200 seconds.
If the value of DurationInSeconds is 0 (zero) or if the first delivery attempt takes longer than the current value, Kinesis Data Firehose does not retry.

Look at the official documentation for more insights:
https://docs.aws.amazon.com/firehose/latest/APIReference/API_RedshiftRetryOptions.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-retryoptions.html

Aman_A · Answer

Hello,

Each Amazon Data Firehose destination has its own data delivery failure handling. You can refer following documentation-

[] Understand data delivery in Amazon Data Firehose  - Handle data delivery failure - https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#retry

Depending on the destination you can specify a retry duration when creating a Firehose stream and this feature is independent of whether you have enabled Dynamic Partitioning or not for your Firehose stream.

firehose retry behaviors

Question got answered before

Confusion

Add your answer

Relevant content