Kinesis Firehose kafka headers
Is there an option to collect the Kafka headers when consuming a Kafka topic in kinesis firehose? If not I think it would be a great improvement.
- Newest
- Most votes
- Most comments
Well, upon this documentation, it mentions that you can set the serialization settings once you choose MSK as a source and S3 as a destination:
"An Apache Kafka record carries three fields: value, key and optional headers. With Amazon MSK as the source of the delivery stream, Kinesis Data Firehose ingests the three Kafka record fields - value, key and headers - and delivers them to Amazon S3 as separate files. The Kafka record value is delivered by default. Kafka key and Kafka header are optional and delivered only if you select them. "
Honestly, I haven't tested this feature before.
Perhaps if you mean something like Dynamic partitioning, it is still not available for MSK source. I hope this can be added in the future. Alternatively, this possibly can be achieved through a Lambda attached to the Firehose.
Check these resources that may contain additional details:
Dynamic partitioning could be handy but what I mean is collecting from kafka the message headers. This is already supported when streaming kakfa topics from Glue by example. I need to collect/store also those keys.
Relevant content
- asked 2 years ago
- asked 10 months ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 5 months ago
- AWS OFFICIALUpdated a year ago
I will have a look. Honestly I love the Glue Kafka connection approach that add a field to the message body when you ask to "includeHeaders".
Btw this is probably not supported from the console. I cannot see any selectable option to ask for key and headers delivering to S3. Do you?