retry enabling Data lost issue for Firehose

Question

Hey, It been said from [AWS document](https://docs.aws.amazon.com/firehose/latest/dev/troubleshooting.html) that:
> If Firehose encounters errors while delivering or processing data, it retries until the configured retry duration expires. If the retry duration ends before the data is delivered successfully, Firehose backs up the data to the configured S3 backup bucket. If the destination is Amazon S3 and delivery fails or if delivery to the backup S3 bucket fails, Firehose keeps retrying until the retention period ends

It sounds to me that during the retention period of Kinesis Stream, any failure from firehose and Lambda that processes the data will be auto re-processed. Is that statement correct? Is there any configuration I have to make for enabling this feature?

Also. I created a firehose instance manually for testing. I enabled data transform function to convert Json file to Parquet file and no Lambda function associated to that. 
1. I purposely create a invalid schema for data conversion. 
2. Secondly, I manually created a data item with version 15. Of course, the data delivery failed with code Error DataFormatConversion.InvalidSchema.
3. Then I revised the schema to make sure it will go through the data conversion and make it delivered. 
4. I did see there's a data point from `Records delivered to Amazon S3 (Sum)` metric that indicate the version 14 was delivered to S3 (Since the version 15 should be the only change I made during that time)
5. But when I check the S3 destination. I could find such a record. I even used a crawler to crawl the data from S3 then query from Amazon Athena. I couldn't find the data item with version 14.

Why would that data lost issue happen? Did that mean that firehose never re-process the failure records? Or Did I miss anything to enable the retry?

Answer

=>In response to your question:

=>The statement from the AWS documentation is correct. During the retention period of the Kinesis stream, any failures encountered by Firehose, or the associated Lambda function will trigger automatic retries until the data is successfully delivered or the retention period ends. No additional configuration is required to enable this feature.

=>Regarding the issue you encountered with data loss:
=>It seems that the failure records were not successfully reprocessed by Firehose.

=>Check if the retry duration configured for Firehose is sufficient to allow for multiple retries within the retention period.

=>Ensure that the S3 destination bucket has the necessary permissions for Firehose to write data to it.

=>Double-check the data transformation function and ensure that it is correctly configured to handle the data format conversion errors.

Answer

Hello Andrew,

The retry mechanism in Kinesis Firehose is a default feature and you need not change any configuration to enable it.

Having said that, each Amazon Data Firehose destination has its own data delivery failure handling.

Data delivery to your S3 bucket might fail for various reasons. For example, the bucket might not exist anymore, the IAM role that Amazon Data Firehose assumes might not have access to the bucket, the network failed, or similar events. Under these conditions, Amazon Data Firehose keeps retrying for up to 24 hours until the delivery succeeds. The maximum data storage time of Amazon Data Firehose is 24 hours. If data delivery fails for more than 24 hours, your data is lost.

[+] Amazon Data Firehose Data Delivery - Data Delivery Failure Handling - https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#retry

Regarding the queries related to the Firehose instance which you created for testing purposes, I would not be able to answer since we require details that are non-public information. I kindly request you to open a support case with AWS using the following link.

https://console.aws.amazon.com/support/home#/case/create

retry enabling Data lost issue for Firehose

Relevant content