how to create an index in an OpenSearch with multiple vector fields?

Question

based on the documentation , it is possible to create an aws opensearch index with multiple vector fields ( sample below ) . could I simply add another vector field ?

```
PUT /awesome_index
        {
          "settings": {
            "index.knn": true
            "number_of_shards": 2,
             "number_of_replicas": 1
          },
          "mappings": {
            "properties": {
              "vector_field_1": {
                "type": "knn_vector",
                "dimension": 1024,  
                "method": "hnsw"   
              }, 
             "vector_field_1" : {
              ....
              }
            }
          }
        }
```

M Zubair Bin Ramzan · Accepted Answer

Yes, you can add multiple vector fields to an OpenSearch index, but there are a couple of issues with your example that need to be fixed. The key points are:

Each field must have a unique name: You can't use the same name ("vector_field_1") for multiple fields. You need to provide unique names for each vector field.

The vector type and configuration for each field (e.g., "knn_vector") will be the same or similar, but the dimensions and other parameters (e.g., method) can be different depending on your use case.

Here’s an updated example of how to create an index with multiple vector fields:

PUT /awesome_index
{
  "settings": {
    "index.knn": true,
    "number_of_shards": 2,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "vector_field_1": {
        "type": "knn_vector",
        "dimension": 1024,  
        "method": "hnsw"
      },
      "vector_field_2": {
        "type": "knn_vector",
        "dimension": 512,  
        "method": "hnsw"
      },
      "vector_field_3": {
        "type": "knn_vector",
        "dimension": 256,  
        "method": "hnsw"
      }
    }
  }
}
Explanation:
index.knn: This setting enables KNN search for the index.

number_of_shards: Defines the number of primary shards. Adjust this depending on your data volume.

number_of_replicas: Defines the number of replica shards for high availability.

vector_field_1, vector_field_2, vector_field_3: These are your vector fields. Each has a unique name and a defined dimension (e.g., 1024, 512, 256) and a method for the KNN search (hnsw in this case).

Steps:
Ensure Unique Names: Ensure each vector field has a unique name (vector_field_1, vector_field_2, etc.).

Set Dimensions: Specify the correct dimension value for each vector field. The dimension must match the dimensionality of the vectors you're indexing.

Define Search Method: You can choose between hnsw, flann, or other KNN search methods depending on your requirements.

Once you run this PUT request, the index will be created with multiple vector fields, and you can use these fields in your KNN queries accordingly.

Query Example:
When querying this index with KNN, you can specify which vector field to use. For example, to search using vector_field_1, you might use something like:

POST /awesome_index/_search
{
  "query": {
    "knn": {
      "vector_field_1": {
        "vector": [0.1, 0.2, ..., 0.1024], 
        "k": 10
      }
    }
  }
}
This would search vector_field_1 and return the top 10 nearest neighbors based on the query vector. You can do the same for other vector fields as needed.

regards,
M Zubair
https://zeonedge.com

how to create an index in an OpenSearch with multiple vector fields?

Relevant content