Opensearch : cluster is red and never recover -



What should we do when the Opensearch cluster is red and never recover for 3 days, with as notification message: "Your cluster status is red. We have started automatic snapshot restore for the red indices. No action is needed from your side. Red indices .opendistro-ism-config". ?

.opendistro-ism-config is an admin index, we don't have the rights to recover it from a snapshot.

It seems that our data are still arriving on our index. But we fail to access the "policies" menu , with "[search_phase_execution_exception] all shards failed" as error message.

In the Opensearch Dev Tools Console ,with a "GET _cluster/allocation/explain" we can see more detail of the index status : "index" : ".opendistro-ism-config", "shard" : 4, "primary" : true, "current_state" : "unassigned", "unassigned_info" : { "reason" : "CLUSTER_RECOVERED", "at" : "2023-11-20T12:56:49.906Z", "last_allocation_status" : "no_valid_shard_copy" }, "can_allocate" : "no_valid_shard_copy", "allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt",

and with :

POST _snapshot/cs-automated-enc/2023-11-20t10-33-38.*****ID/_restore{ "indices": ".opendistro-ism-config" } we get : { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "no permissions for [] and User [name=, backend_roles=[], requestedTenant=]" } ], "type" : "security_exception", "reason" : "no permissions for [] and User [name=, backend_roles=[], requestedTenant=]" }, "status" : 403 }

Last info, in the AWS Health Dashboard in the "Log event" tab we saw the following message concerning our Opensearch stack : "OpenSearchService domain unavailable" - Issue - Status Closed, but actually the stack never recovered from the red status.

asked 6 months ago305 views
1 Answer

In this case I would raise a ticket with AWS Support and have them take a look at this issue to understand why this Index is in this state and to get it recovered for you.

Barry M
answered 6 months ago
  • Thank you for your feedback, but without an AWS Support subscription, it seems that we can't create a ticket. That is the main reason why I've posted the question, in this "Re:Post" channel. I'm sorry that I forgot to mention it in in my issue description. Do you know a way to create a ticket for this kind of issue, to AWS support without AWS Support subscription ?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions