How does OpenSearch Serverless support 1TiB of index data?

Kindly find the answers to your queries below:

1. Could you explain why 960GB only requires 8 OCUs? Don’t we need the standby OCUs for search? Also how about the OCUs for data ingestion?

Firstly, I would like to mention that Opensearch serverless has Indexing and Search OCUs. When you create your first collection, OpenSearch Serverless instantiates a total of four OCUs (two for indexing and two for search). These OCUs always exist, even when there's no indexing or search activity [1].

Further, the number of OCUs required for an OpenSearch Serverless cluster depends on the specific use case and workload. There is no fixed or standard number of OCUs that can be defined for all use cases. The number of OCUs required for a cluster depends on factors such as the size of the data set, the frequency of data updates, the number of concurrent search queries, the complexity of the search queries, Storage retained in Amazon S3 [2].

Thus, as an approximate if you are using Search collection then, 960GB of data can take 8 OCUs for Indexing which is ( 960 GB of data / 120 GB of data per OCU ) and 8OCU for search as well. However, this can increase/decrease according to your usage as mentioned in [2]. If you have high search request then the capacity can scale up to accommodate data, it can’t currently scale up search replicas. OpenSearch Serverless functions with two search replicas running in active-active mode.

Kindly note, the formula ( 960 GB of data / 120 GB of data per OCU ) is just a generic formula to calculate the approximate number of OCUs required but it may vary according to the usecase as stated above.

Thus, to check how many OCUs are required AWS recommend users to create a test collection and do a small testing with the workload that you think that may come. However, if there is high or low workload the OCUs will increase or decrease accordingly.

2. If the 960GB data is replicated across the 8 OCUs for redundancy, do you mean one OCU will have much more than 120GB data? This sounds conflict with the 120GB limit.

As mentioned above, the number of OCUs required for an OpenSearch Serverless cluster may vary depending on the workload, and it is difficult to provide an exact number. If the workload includes a high volume of search requests, the capacity can scale up to handle the data [2]. The data can be replicated in search OCUs ephemeral storage.

Further more, the requirement of OCU also depends upon if you are using Search collection or time series collection.

I understand that you would like to get cost estimation and a good idea how OCUs work so that you can test how many OCUs will be required for different collection. Thus, here you can manage capacity for your collections and to control costs, you can specify the overall maximum indexing and search capacity for the current account and Region, and OpenSearch Serverless scales out your collection resources automatically based on these specifications [1].

• Maximum indexing capacity – OpenSearch Serverless can increase indexing capacity up to this number of OCUs.

• Maximum search capacity – OpenSearch Serverless can increase search capacity up to this number of OCUs.

References

[1] Managing capacity limits for Amazon OpenSearch Serverless - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-scaling.html

[2] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-scaling.html

Please find below in-line the response:

Each OCU includes enough hot ephemeral storage for 120 GiB of index data[1]. Using this, you will need approx. 8 OCUs to support a full-text search workload for a 960GB collection.

The index data is cached across all the OCUs in a collection. Each OCU contains a portion of the index data, and the data is replicated across the OCUs for redundancy. This ensures that the data is readily available for processing and querying.

When queries are submitted, the OpenSearch serverless will automatically scale up and down based on the current workload. The queries will not have to wait for all the OCUs to be ready. The scaling is in response to query demand, so additional OCUs will be added dynamically and will become available for processing queries as soon as they are ready. This ensures that query performance is maintained as the query volume changes over time.

References: [1] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-scaling.html