We recently changed our cluster configuration (increasing the size of data node instances), which resulted in a blue/green deployment, which appeared to succeed. OpenSearch is reachable, ingesting data and accepting queries directly via the HTTP API, however we are unable to access the Kibana dashboard at all. We don't believe any individual nodes are under duress, and instead suspect that something may have gone wrong as part of provisioning new nodes to host the cluster and copying over the previous cluster's shard data.
From a bit of research we have done, it seems that the suggestion is to "reboot the cluster", which there is no option in the AWS Console (or via the CLI) that I can see, and apparently would have to be triggered internally by AWS. We are unable to get directly in touch with support for OpenSearch via the Support Center as we are currently on the 'basic' plan, and I don't think it is reasonable to expect paying an increased amount of support solely to get AWS to perform a fundamental operation (rebooting) as part of a managed service.
Is there potentially a way to get our Kibana dashboards accessible again, preferably if it could be actioned by us? If it helps, out of curiosity, I tried to see if we could perform an OpenSearch version upgrade (2.5 -> 2.9) to see if that could trigger a 'reboot' by the backdoor, however it stated that "All kibana nodes are OOS in elasticsearch" in the error logs that appeared after checking for upgrade eligibility. I don't know if this provides any additional context that might show the path forward here.
AWS OFFICIALUpdated 10 months ago
