AWS OpenSearch SearchOCU keeps hitting the max limit
I've created 5 Bedrock Knowledge bases, with a collective total file size around 278 GiB.
I've had no issues before this, but 2 weeks ago I've started to hit the SearchOCU limit reached error. My use case is pretty simple, we have a Lambda which will call AWS Bedrock Retrieve API to get our relevant data and that's it.
Usage wise barely reached 100 request per day. Right now I've set a limit of 50 SearchOCU and thankfully it's staying around 28 SearchOCU usage. But I'm not sure when it'll increase again and it's adding unnecessary extra costs in our account.
I checked SearchRequestRate metric and usage is < 100 at any given time. Refer to attached image:
Hence I have 2 question:
- What operations uses SearchOCU in terms of Bedrock knowledge base, because we used OpenSearch exclusively to query Bedrock
- What other metrics can I use to determine what might've cause the SearchOCU limit reach issue?
- Newest
- Most votes
- Most comments
My OpenSearch domain exhibited unexpected Search Capacity Unit (SearchOCU) scaling behavior correlated with the number of Collections, even with minimal query activity. After deleting a large number of Collections, retaining only critical Collections totaling less than 5GB, the SearchOCU count decreased to 2. Previously, with a significantly higher number of Collections, the SearchOCU count was substantially inflated, despite low query volume.
This observation suggests that the sheer presence of a large number of OpenSearch Collections, independent of active search queries, influences SearchOCU consumption. While I understand the impact of query load on scaling, the mechanism by which the number of Collections drives SearchOCU inflation remains unclear.
answered 2 years ago
To address your concerns about the SearchOCU usage in AWS OpenSearch Serverless with Bedrock Knowledge Bases, let's break down the answers to your questions:
-
SearchOCU usage in Bedrock Knowledge Base: When you use Bedrock Knowledge Base, which is built on top of OpenSearch Serverless, SearchOCUs are primarily used for search operations. Even if you're not directly querying OpenSearch, the Bedrock Retrieve API likely uses SearchOCUs to process and retrieve relevant data from your knowledge bases. The large collective file size of 278 GiB across your 5 knowledge bases could be contributing to the higher SearchOCU usage, as larger datasets may require more computational resources to search efficiently.
-
Metrics to determine SearchOCU limit issues: While you've already checked the SearchRequestRate, which is showing low usage, there are other metrics you can monitor to get a better understanding of your SearchOCU utilization:
- SearchOCU: This is the primary metric to watch. It shows the number of OCUs used for search operations.
- IndexingOCU: Although your issue is with SearchOCU, monitoring this can give you a complete picture of your resource usage.
- SearchableDocuments: This metric can help you understand if there's been a significant increase in the number of searchable documents, which could impact SearchOCU usage.
- TotalFieldCount: A high number of fields across your indices can impact search performance and OCU usage.
It's worth noting that OpenSearch Serverless maintains a minimum of 2 OCUs for search even when there's no activity. The system scales up based on various factors, including data size and query complexity, not just request rate.
To manage costs and understand usage patterns:
- Monitor these metrics using Amazon CloudWatch and set up alarms to notify you when approaching thresholds.
- Analyze your query patterns and optimize if possible.
- Consider reviewing your data retention policies and archiving older, less frequently accessed data if appropriate.
- Evaluate if your current capacity limits are appropriate for your workload.
If the issue persists without a clear reason, it may be worth contacting AWS support for a more in-depth analysis of your specific use case and configuration.
Sources
Managing capacity limits for Amazon OpenSearch Serverless - Amazon OpenSearch Service
OpenSearch Serverless OCU usage too high | AWS re:Post
opensearch cost increase drastically in last two weeks from $5.76 to $41.99 per day | AWS re:Post
answered 2 years ago
Relevant content
asked 8 months ago
