Choosing Between "Direct" and "Buffering" Delivery Modes in Amazon Kinesis Firehose: Trade-offs and Considerations

I have found the following refs

https://stackoverflow.com/questions/62587569/kinesis-data-firehose-source-direct-put-vs-kinesis-data-stream https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html https://docs.aws.amazon.com/firehose/latest/dev/create-configure.html

The concepts of "direct" and "buffering" delivery modes are not explicitly defined in Amazon Kinesis Data Firehose. However, based on your question, it appears that you are referring to the difference between Direct PUT and using a Kinesis Data Stream as a source, or you may be referring to the buffering options available when dealing with data transformation and backup.

Direct PUT vs. Kinesis Data Stream: The choice between these largely depends on your use case. If you only aim at storing (and possibly transforming) your records in supported storage destinations, Direct PUT to Firehose is simpler. With Direct PUT, you manually inject records into Firehose using put-record or put-record-batch API calls. Firehose handles the scaling and throughput, but your records may experience some delay due to timeout and buffering. However, if you require real-time processing and have other consumers of your data stream, using a Kinesis Data Stream may be more appropriate, despite the added complexity of managing key partitioning and throughput​.

Buffering in Data Transformation: When data transformation is enabled, Kinesis Data Firehose buffers incoming data, which can range between 0.2 MB to 3MB. The buffered batch is then passed to the specified Lambda function for transformation. The transformed data is sent to the destination when the specified destination buffering size or buffering interval is reached, whichever happens first​.

Buffering in Backup: Kinesis Data Firehose uses buffering before delivering data to Amazon S3 for backup. You can choose a buffer size of 1–128 MiBs and a buffer interval of 60–900 seconds. The delivery to S3 is triggered by whichever condition is satisfied first. If data delivery to the destination falls behind data writing to the delivery stream, Kinesis Data Firehose dynamically increases the buffer size to ensure that all data is delivered to the destination​.