Kinesis data Firehose Data Transformation and Compression

Question

Hi all,

We are working on a solution based on **CloudWatch Logs Destinations** and **Kinesis Firehose** to be able to centralize **CloudWatch Logs** coming from **multiple Application Accounts** inside our **Organization**  in a **Centralized Log Account** as shown in the attached image below and described also [here](https://dev.to/aws-builders/cross-account-log-data-sharing-using-kinesis-data-firehose-4e9b) :

![Enter image description here](/media/postImages/original/IMp39YSxpyS1mrrVzL-XrlXw)

We are using a **Lambda function** triggered by the **Firehose Stream** for **Data Transformation** before sending data to the final destination **S3 Bucket**.

The Lambda should **unzip** the recieved data, process it and send it back to the **Firehose Stream** "Based on Firehose Lambda Blueprint"

Everything is working as expected, except for the **Kinesis Firehose Compression**,
On the Firehose Destination Settings, We've already **enabled Compression** of data records in **GZIP** format but what we are recieving on the S3 Bucket are **uncompressed** files with **.gz extension** !!

![Enter image description here](/media/postImages/original/IM7gRimfnWTC2YkG7QTDGQ0A)

any idea on how to resolve this issue ?

should the Lambda compress the data records after processing them and before sending them back to the Firehose Stream and how ?

Thanks in advance

Accepted Answer

I managed to figure out what wase the issue,

Kinesis Firehose while writing objects to S3, it adds a key/value pair, \"Content-Encoding=gzip” to the metadata of the objects.

When we request to download the object from S3 on a browser(UI), this key/value is added to the response header which causes the browser to automatically decompress the object. Latest browsers come with this built-in feature to decompress such files upon download.

Hence, the files stored in the S3 bucket are in correct format with compressed values but when we download it, browser automatically decompresses it and it is an expected behaviour for Kinesis Firehose delivery to S3 with GZIP compression enabled.

Kinesis data Firehose Data Transformation and Compression

関連するコンテンツ