Hi,
What should we do when the Opensearch cluster is red and never recover for 3 days, with as notification message:
"Your cluster status is red. We have started automatic snapshot restore for the red indices. No action is needed from your side.
Red indices .opendistro-ism-config". ?
.opendistro-ism-config is an admin index, we don't have the rights to recover it from a snapshot.
It seems that our data are still arriving on our index. But we fail to access the "policies" menu , with "[search_phase_execution_exception] all shards failed" as error message.
In the Opensearch Dev Tools Console ,with a "GET _cluster/allocation/explain" we can see more detail of the index status :
"index" : ".opendistro-ism-config",
"shard" : 4,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2023-11-20T12:56:49.906Z",
"last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "no_valid_shard_copy",
"allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt",
and with :
POST _snapshot/cs-automated-enc/2023-11-20t10-33-38.*****ID/_restore{
"indices": ".opendistro-ism-config"
}
we get :
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "no permissions for [] and User [name=, backend_roles=[], requestedTenant=]"
}
],
"type" : "security_exception",
"reason" : "no permissions for [] and User [name=, backend_roles=[], requestedTenant=]"
},
"status" : 403
}
Last info, in the AWS Health Dashboard in the "Log event" tab we saw the following message concerning our Opensearch stack :
"OpenSearchService domain unavailable" - Issue - Status Closed, but actually the stack never recovered from the red status.
Thank you for your feedback, but without an AWS Support subscription, it seems that we can't create a ticket. That is the main reason why I've posted the question, in this "Re:Post" channel. I'm sorry that I forgot to mention it in in my issue description. Do you know a way to create a ticket for this kind of issue, to AWS support without AWS Support subscription ?