How to combine OpenSearch Serverless indexes with wildly different fields and different embeddings?

Combining OpenSearch Serverless indexes with different fields and embeddings is possible, but it requires careful consideration and may involve some trade-offs. Here's how you can approach this:

1. Create a new index: Instead of trying to merge the existing indexes directly, create a new index that can accommodate the fields from both sources.

2. Define a unified schema: In the new index, define a schema that includes all the fields from both sources. You may need to rename some fields to avoid conflicts and ensure consistency.

3. Reindex the data: Use OpenSearch's reindex API to copy data from both source indexes into the new unified index. During this process, you can map the fields from the source indexes to the corresponding fields in the new index.

4. Handle the embeddings: Since both indexes use 1536-dimensional vectors, you can create a single vector field in the new index to store embeddings from both sources. However, be aware that mixing embeddings from different models (Bedrock and your custom API) in the same field may impact search quality.

5. Address the metadata issue: For the unfilterable metadata from the Bedrock index, you have a few options:

   • Store it as-is, accepting that it won't be filterable.
   • Extract key information from it and store it in separate, filterable fields.
   • Use OpenSearch's ingest pipelines to process and transform the metadata during ingestion.

6. Adjust querying: When querying the combined index, you'll need to account for the differences in field names and potentially adjust your search algorithms to handle the mixed embeddings.

Regarding the unfilterable nature of the first index's metadata, it doesn't make combining the indexes impossible, but it does limit your ability to filter on that specific field. You can still search and retrieve the content, but you won't be able to use it for filtering or aggregations.

Keep in mind that while this approach allows you to query the combined knowledge, the difference in embeddings and field structures may affect the consistency and accuracy of your search results. You might need to experiment with different querying strategies to get the best results from your combined index.
Sources
Exploring Embeddings and LLMs: Amazon Titan Embedding V2, Claude 3.5 and OpenSearch Serverless | AWS re:Post
Create a knowledge base with an OpenSearch Serverless collection | AWS re:Post
Using an OpenSearch Ingestion pipeline with OpenSearch - Amazon OpenSearch Service