504 Errors after moving from OpenSearch Cluster to a Serverless Collection

Question

Hi,
we moved our application from an OpenSearch Cluster to a serverless collection and we get a lot of errors since then.
Most of the errors are from our search requests: `Request failed: [http_exception] server returned 504`

The errors usually appear in a period over 10 minutes until our instances crash due to the failed health check, because the collection is not responding and then everything works fine for some hours.

We set our Search Capacity OCUs to 34 and even during this error period only 16 OCUs are used.

We don't really know how to debug this and what the actual problem this, because we don't get many information from the collection.

We use the Java OpenSearchClient with the AwsCrtHttpClient and are not really sure about the configuration, because there no real documentation about best practices for values like `maxConcurrency`

re:Post Agent · Answer

When transitioning from an OpenSearch cluster to a serverless collection, 504 Gateway Timeout errors typically indicate that the service is taking too long to respond to your requests.

There are several potential causes and solutions to consider:

1. **Index Creation Delay**: When creating indexes in OpenSearch Serverless, there's typically a delay of up to 30 seconds before the index becomes fully operational. If you're sending data immediately after index creation, you might encounter errors including 500-level errors. Implementing a retry mechanism with exponential backoff in your application can help manage this.

2. **Connection Configuration**: Your Java OpenSearchClient with AwsCrtHttpClient might need tuning. While there's limited documentation on optimal `maxConcurrency` values, you might need to adjust this setting based on your workload. Consider implementing connection pooling and increasing timeouts if your queries are complex.

3. **Query Optimization**: Review your search queries for complexity. Overly complex queries might time out, especially if they involve large datasets or complex aggregations.

4. **OCU Utilization**: Even though you've set 34 OCUs for search capacity and only 16 are being used, the issue might not be related to raw capacity but rather to how the requests are distributed or how the collection is handling specific query patterns.

5. **Collection Size Limits**: Ensure you're not exceeding the collection size limits. OpenSearch Serverless supports 10 TB per time-series collection and 1 TB per search or vector collection.

For debugging:
- Monitor CloudWatch metrics for your collection, particularly `IngestionRequestErrors` and `IngestionRequestSuccess`
- Implement proper error handling in your code with retry logic
- Consider implementing a queue system if you're creating multiple indexes in quick succession
- Test with simpler queries to isolate if specific query patterns are causing the timeouts

If these steps don't resolve the issue, you might need to contact AWS Support for more detailed troubleshooting of your specific collection configuration.  
**Sources**  
[Opensearch Serverless scale related questions | AWS re:Post](https://repost.aws/questions/QUee1ufV2-Ti6JFnqBSvlfNg/opensearch-serverless-scale-related-questions)  
[AWS OpenSearch Serverless index creation delay | AWS re:Post](https://repost.aws/questions/QUukIQ9nnDSRSHXvdDSqQO2w/aws-opensearch-serverless-index-creation-delay)  
[OpenSearch Configuration Change Stuck at 20% and 503 Error Responses | AWS re:Post](https://repost.aws/questions/QUlCZCRm92RIWbACsjlmbh5g/opensearch-configuration-change-stuck-at-20-and-503-error-responses)

504 Errors after moving from OpenSearch Cluster to a Serverless Collection

Add your answer

Relevant content