What is best practice for updating a document in OpenSearch Serverless

Question

When previously using Elastic, we would construct our id for a document which enables us to easily update by id - the id essentially being a a guaranteed unique string based on the source tenant+system+sourceId+type.

When creating a record in an index in OpenSearch Serverless (Vector index), I believe it is not possible to specify the id. Instead, OpenSearch provides its own id.

What is best practice when a specific document needs updating in the index?

1. Are we missing a method for specifying our own unique id so we can just PUT / POST the updated document?
2. Is it best practice to record the id somewhere (Dynamo or wherever) as a lookup table?
3. Or is the "Standard" way of updating a document to search for the document then update based on result? This will give us the knock on effect of high search traffic just to update the index.
4. Something else!

More info:

re:Post Agent · Answer

When working with OpenSearch Serverless, particularly with vector search collections, the approach for updating documents differs from traditional Elasticsearch or OpenSearch practices. Here's what you need to know:

1. For vector search collections in OpenSearch Serverless, you cannot index by custom document ID or use upsert requests. This is a limitation specific to this type of collection.

2. The best practice for updating documents in this scenario depends on your specific use case, but here are some approaches to consider:

a. Use the OpenSearch-provided ID: When you index a document, OpenSearch Serverless will generate an ID. You should store this ID along with your application-specific identifiers (like tenant+system+sourceId+type) in a separate database, such as Amazon DynamoDB. This allows you to maintain a mapping between your logical identifiers and the OpenSearch document IDs.

b. Update by document ID: Once you have the OpenSearch-generated ID (retrieved from your mapping database), you can use it to update the document directly. This is more efficient than searching for the document first.

c. Avoid frequent updates if possible: Vector search collections are optimized for search operations rather than frequent updates. If your use case involves frequent updates, you might want to consider using a different collection type or restructuring your data model.

3. Searching for a document and then updating based on the search result is generally not recommended as the primary update strategy, especially for frequent updates. As you correctly pointed out, this would lead to increased search traffic and potential performance issues.

4. Consider your update frequency and volume:
   - For infrequent updates, the lookup table approach (storing OpenSearch IDs in DynamoDB) can work well.
   - For high-volume updates, you might want to batch your updates or reconsider your data model to minimize the need for updates in the vector search collection.

Remember, OpenSearch Serverless applies eventual consistency for updates. When you update a document, the old version is marked as deleted, and a new document is added. The old version doesn't disappear immediately but becomes inaccessible. OpenSearch cleans up deleted documents in the background as you continue to index more data.

In summary, the recommended approach is to maintain a mapping of your logical identifiers to OpenSearch-generated IDs, use these IDs for updates, and design your data model and update strategy to align with the strengths of vector search collections in OpenSearch Serverless.  
**Sources**  
[What is Amazon OpenSearch Serverless? - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html)  
[Community | Unleashing CRUD Prowess: OpenSearch and Golang, a Dynamic Duo](https://community.aws/content/2emFiyrjxYN6ofogVHDtxVGN4nj/unleashing-crud-prowess-opensearch-and-golang-a-dynamic-duo)

What is best practice for updating a document in OpenSearch Serverless

Add your answer

Relevant content