how to create/edit opensearch instance via console ?

Question

I want to create an aws opensearch cluster to save some text embeddings. Most of the settings are defaulted,   'r5.large.search' for index type, general purpose (ssd) gp2 and size to 50GiB for EBS volume. 
First question I have is , can I edit these values later , if I need more capacity and better latency. say if I have 500, 000 documents , each document approximately 50 kilobytes, is there an easy estimation on the EBS size and the instance type selection for moderate performance.

After that, I created an index (sample code below) . based on the example , I set number of shards  to 2 and replica to 1. also,  based on the docs , I want to search K nearest neighbors , so   I set the index.knn to true.
I assume these settings for particular index are not editable once it is created, Is that a correct assumption?
other parameters like ef_construction and space_type (see sample below)  => 
is there a specific values , I need for KNN search or is there a general rule of thumb for these values given my use case and number of documents ,I'm trying to index/search?
```
 "properties": {
      "my_vector": {
        "type": "knn_vector",
        "dimension": 4,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "nmslib",
          "parameters": {
            "ef_construction": 128,
            "m": 24
          }
        }
      }
```

create index 
```
 PUT /awesome_index
        {
          "settings": {
            "index.knn": true
            "number_of_shards": 2,
             "number_of_replicas": 1
          },
          "mappings": {
            "properties": {
              "vector_field": {
                "type": "knn_vector",
                "dimension": 1024,  
                "method": "hnsw"   
              }
            }
          }
        }

```

Jainam · Accepted Answer

**Editing Cluster Configuration:**
You can modify certain configurations of an Amazon OpenSearch Service cluster after creation:
- **Change instance type:** Scale up or down based on performance needs.
- **Modify EBS volume size:** Increase the size of EBS volumes.

**Estimation for 500,000 documents (50 KB each):**
- **Total data size:** 25 GB
- **EBS volume:** Start with 50 GiB, scale up if needed.
- **Instance type:** Start with `r5.large.search`, scale to `r5.xlarge.search` for better performance.

**Index Settings:**
Once created, you cannot change:
- `number_of_shards`
- `number_of_replicas`
- `index.knn`

Create a new index and reindex data if changes are needed.

**KNN Parameters:**
- **`ef_construction`:** Start with 128 for a balance between build time and query performance.
- **`m`:** Use a value between 16 and 32.
- **`space_type`:** Use `l2` for Euclidean distance.

**Sample Index Creation:**

```json
PUT /awesome_index
{
  "settings": {
    "index.knn": true,
    "number_of_shards": 2,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "vector_field": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "parameters": {
            "ef_construction": 128,
            "m": 24
          }
        }
      }
    }
  }
}
```

re:Post Agent · Answer

Creating and editing an Amazon OpenSearch instance via the console is a straightforward process. Here's how you can do it and address your questions:

1. Creating the OpenSearch domain:
You can create an OpenSearch domain through the AWS console by navigating to the Amazon OpenSearch Service section and clicking "Create domain". You've mentioned using 'r5.large.search' instance type with 50GiB EBS volume (gp2).

2. Editing values later:
Yes, you can edit these values later. Amazon OpenSearch Service allows you to modify your domain configuration after creation. You can change instance types, increase storage, and adjust other settings as your needs evolve.

3. Estimating resources for 500,000 documents:
For 500,000 documents of approximately 50KB each, you're looking at about 25GB of raw data. As a general rule, you should allocate 2-3 times this amount for OpenSearch, so your initial 50GB should be sufficient to start. However, for better performance as your data grows, you might want to consider scaling up to a larger instance type (like r5.xlarge.search) and increasing your EBS volume size.

4. Index settings:
Regarding your index creation, you're correct that some settings cannot be changed after the index is created. This includes the number of primary shards. However, you can change the number of replicas later.

5. KNN-specific settings:
For KNN search, setting "index.knn" to true is correct. The other parameters like "ef_construction" and "space_type" depend on your specific use case:

- "space_type": "l2" is commonly used for Euclidean distance, which is often suitable for text embeddings.
- "ef_construction": 128 is a reasonable starting point. Higher values can improve search quality but increase indexing time.
- "m": 24 is also a reasonable default. It affects the number of connections per node in the graph.

These values often require tuning based on your specific data and performance requirements. Start with these defaults and adjust based on your observed performance.

Remember, while you can't change these settings for an existing index, you can create a new index with adjusted settings and reindex your data if needed.

Lastly, when working with text embeddings, ensure your "dimension" parameter matches the size of your embedding vectors. In your example, it's set to 1024, which should align with your embedding model's output.

As you scale and optimize your OpenSearch deployment, keep monitoring your cluster's performance and adjust these settings accordingly.  
**Sources**  
[Community | Search and Analyze data in Amazon OpenSearch using SQL](https://community.aws/content/2iCi0laewojPi0LQcnX3MXn5wR9/search-analyze-using-sql-with-amazon-opensearch)  
[Create an Amazon OpenSearch Service domain - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsgcreate-domain.html)  
[Using the AWS SDKs to interact with Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configuration-samples.html)

how to create/edit opensearch instance via console ?

Add your answer

Relevant content