How to create and configure an Open Search Serverless index via API?

Creating and configuring an OpenSearch Serverless index via API is possible, but there are some limitations on what can be configured compared to managed OpenSearch domains.

To create an index in an existing OpenSearch Serverless collection, you can use the OpenSearch API. You'll need to establish a secure connection to the specified OpenSearch Serverless collection endpoint using an appropriate client library (such as opensearch-go for Go, or similar libraries for other languages).

You can create an index and specify some configurations like the number of shards and basic mappings. For example, you can set the precision and dimensions for vector fields. Here's a general approach:

1. Connect to your OpenSearch Serverless collection endpoint using a client library.
2. Use the Indices Create API to create a new index with your desired settings.

However, it's important to note that OpenSearch Serverless abstracts away many of the low-level cluster configurations. Some of the specific settings you mentioned (engine, distance type, M, ef_construction) are typically associated with the underlying vector search engine and are not directly configurable in OpenSearch Serverless.

OpenSearch Serverless automatically handles indexing and storage. When you write data to a collection, the service distributes it to indexing compute units, which ingest the incoming data and move the indexes to S3. This process is managed automatically by the service.

While you can't directly configure low-level engine parameters like M and ef_construction, you can still create vector fields and specify their dimensions and precision. The underlying optimizations and configurations are handled by the serverless infrastructure to provide optimal performance.

If you require more granular control over these specific engine parameters, you might need to consider using a managed OpenSearch domain instead of the serverless option. The serverless option trades some configuration flexibility for easier management and automatic scaling.
Sources
Ingesting data into Amazon OpenSearch Serverless collections - Amazon OpenSearch Service
How is the indexing process taken care of in opensearch serverless? | AWS re:Post