Opensearch : cluster is red and never recover -

Hi,

What should we do when the Opensearch cluster is red and never recover for 3 days, with as notification message: "Your cluster status is red. We have started automatic snapshot restore for the red indices. No action is needed from your side. Red indices .opendistro-ism-config". ?

.opendistro-ism-config is an admin index, we don't have the rights to recover it from a snapshot.

It seems that our data are still arriving on our index. But we fail to access the "policies" menu , with "[search_phase_execution_exception] all shards failed" as error message.

In the Opensearch Dev Tools Console ,with a "GET _cluster/allocation/explain" we can see more detail of the index status : "index" : ".opendistro-ism-config", "shard" : 4, "primary" : true, "current_state" : "unassigned", "unassigned_info" : { "reason" : "CLUSTER_RECOVERED", "at" : "2023-11-20T12:56:49.906Z", "last_allocation_status" : "no_valid_shard_copy" }, "can_allocate" : "no_valid_shard_copy", "allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt",

and with :

POST _snapshot/cs-automated-enc/2023-11-20t10-33-38.*****ID/_restore{ "indices": ".opendistro-ism-config" } we get : { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "no permissions for [] and User [name=, backend_roles=[], requestedTenant=]" } ], "type" : "security_exception", "reason" : "no permissions for [] and User [name=, backend_roles=[], requestedTenant=]" }, "status" : 403 }

Last info, in the AWS Health Dashboard in the "Log event" tab we saw the following message concerning our Opensearch stack : "OpenSearchService domain unavailable" - Issue - Status Closed, but actually the stack never recovered from the red status.