How do I resolve an HTTP 503 Service Unavailable error in OpenSearch Service?

Resolution

The OpenSearch Service domain uses a load balancer to distribute incoming traffic to the data nodes. A data node that's overloaded results in an HTTP 503 status code. When an expensive query or incoming traffic overloads a node, the node doesn't have enough capacity to manage any other incoming requests.

Note: You can use the 5xx metric in Amazon CloudWatch to monitor 5xx HTTP status codes.

To resolve the HTTP 503 status code error, provision more compute resources. Or, reduce the utilization of your queries.

Provision more compute resources

Take the following actions:

• Based on performance, switch to a larger instance to scale up your domain for either a primary node or data node. Or, add more nodes to the cluster. For more information, see How can I scale up or scale out an OpenSearch Service domain?
• Use an instance type with an appropriate amount of storage for your workload. For more information, see Choosing instance types and testing.

Note: Blue/green deployment activities might result in a 503 status code. To resolve this issue, wait for the deployment to complete and for the cluster to return to the Active state.

Reduce the resource utilization of your queries

Take the following actions:

• Follow best practices for shard and cluster architecture. For more information, see Get started with Amazon OpenSearch Service: How many shards do I need?
• Reduce the number of concurrent requests to the domain.
• Reduce the scope of your query. For example, if you run a query for a specific time frame, then reduce the date range. You can also configure index patterns on OpenSearch Dashboards to filter the results.
• Don't run select * queries on large indices. Instead, use filters to query a part of the index and search as few fields as possible. For more information see Query and filter context and Search as few fields as possible on the Elasticsearch website.
• Reindex and reduce the number of shards. The more shards that you have in your cluster, the more likely you receive a courier fetch error. Because each shard has its own resource allocation and overheads, a large number of shards can strain your cluster.

Related information

How can I prevent HTTP 504 gateway timeout errors in Amazon OpenSearch Service?

Operational best practices for Amazon OpenSearch Service

Troubleshooting Amazon OpenSearch Service

How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?

How do I troubleshoot high CPU utilization on my Amazon OpenSearch Service cluster?