Opensearch upgrade stuck
We have a single node Opensearch 1.1 cluster. After selecting to upgrade it to 1.2 the cluster has been stuck on "Preparing to process updates". I found out that it is stuck because the default settings for .opendistro- indices are to create 5 primaries and 1 replica shard. As our cluster only consists of a single shards those some of the .opendistro- indices are not unallocated. I created index templates for those indices in question with 1 primary and 0 replicas and deleted some of the .opendistro indices but there are .opendistro-alerting-alerts indices which I can not delete even though I have all the permissions possible.
Also followed this guide but still getting 403 https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-disable-index-alerts/
GET _cat/shards
.opendistro-alerting-alerts 3 r UNASSIGNED
.opendistro-alerting-alerts 1 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts 1 r UNASSIGNED
.opendistro-alerting-alerts 2 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts 2 r UNASSIGNED
.opendistro-alerting-alerts 4 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts 4 r UNASSIGNED
.opendistro-alerting-alerts 0 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts 0 r UNASSIGNED
.opendistro-alerting-alert-history-2022.03.16-1 3 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1 3 r UNASSIGNED
.opendistro-alerting-alert-history-2022.03.16-1 1 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1 1 r UNASSIGNED
.opendistro-alerting-alert-history-2022.03.16-1 2 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1 2 r UNASSIGNED
.opendistro-alerting-alert-history-2022.03.16-1 4 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1 4 r UNASSIGNED
.opendistro-alerting-alert-history-2022.03.16-1 0 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1 0 r UNASSIGNED
DELETE .opendistro-alerting-alert-history-2022.03.16-1 DELETE .opendistro-alerting-alerts { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]" } ], "type" : "security_exception", "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]" }, "status" : 403 }
I would gladly add some additional data nodes to the cluster in order to fix this but this also fails as the cluster is stuck on "Preparing to process updates".
EDIT: I tried deleting the .opendistro indices with "m aster" user but even "m aster" user receives error 403.
If we had premium support we wouldn't be asking for help in here. I was under the impression that this is the place to come for help in case you don't have a paid support plan.
Could you answer why is the "\master" user not able to delete the indices I mentioned? If we'd get those deleted then we'd probably be able to carry on with the upgrade.
Also this site does not allow posting the word "\master".
When the cluster is in processing state, the cluster will try to complete the blue/green deployment change for a limited number of times before getting stuck. Once the cluster has failed the maximum number of retries possible, the cluster will remain in processing state until intervened by AWS internal team. Please reach out to the premium support with the ARN of the cluster to get the cluster back to the active state.
Other recommendation is to please follow the best practices for Amazon OpenSearch Service: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html
Blue/green deployment: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html
Relevant questions
openSearch cannot be deleted as it stuck in "Being Deleted"
asked 5 months agoAWS Opensearch cluster is in "processing"
Accepted Answerasked 4 months agoOpenSearch cluster stuck in creation
Accepted Answerasked 23 days agoRedshift Cluster Resizing stuck at 97%
asked 6 days agoAWS ElasticSearch Stuck Upgrade processing after 3.5 hours
asked 2 years agoUpgrading Aurora MySQL 5.6 to 5.7 on cross region replica stuck in pre-upgrade checks and reboots every 30 minutes
asked 2 months agoOpensearch 3 node cluster's diskspace was upgraded from 500GB to 600GB. The copying of shards to new nodes became stuck.
asked 21 days agoOpensearch upgrade stuck
asked 2 months agoUpgrade ElasticSearch to Opensearch taking 2 months
asked 3 months agoOpenSearch upgrade Stuck at "Check eligibility" for multiple hours
asked 4 months ago