Hi, have you figured out how Firehose handles upsert to OpenSearch? I also need this information for my use case
Firehose Delivery Stream destinations are append-only and in the case of Opensearch, do not support upsert. Firehose will generate a unique ID for each record it streams and use that as the document ID. This cannot be user-configured at this time. If you are an AWS Enterprise Support customer, you can request this feature be added to Firehose by talking with your Solution Architect (SA) or Technical Account Manager (TAM).
One possible short-term solution is to use a Kinesis Stream and trigger a Lambda function to upsert documents to Opensearch using the Opensearch APIs. The Python client would push JSON data to the Kinesis Stream, and rather than having the Lambda function only perform transformations, it would trigger for records in the stream, perform the transformation, and handle upserting to Opensearch.
How to set document id when delivering data from Kinesis Data Firehose to Opensearch indexasked 6 months ago
Streaming data to an HTTP endpoint with Amazon Kinesis Data Firehoseasked 2 months ago
Writing data to Kinesis stream from PySparkAccepted AnswerEXPERTasked 4 years ago
Kinesis Transformation Buffering from Data StreamAccepted Answerasked 4 months ago
IP range of Kinesis Data Firehose custom HTTP endpointasked a month ago
How to set the starting position for a Kinesis Delivery Streamasked 2 months ago
Kinesis data stream - data transfer charges for on-premises consumer/producerAccepted Answerasked 4 years ago
Transferring data from AWS Kinesis data streams to AWS Redshift using AWS Glueasked 2 months ago
Join Streaming Data in Amazon Kinesis Firehose with Data in S3 via Athena using a Data transformation Lambda functionAccepted Answerasked 2 years ago
Multiple Kinesis Data Analytics apps to use the same Kinesis firehose delivery stream as sourceAccepted Answerasked 2 years ago