Opensearch upgrade stuck

Question

We have a single node Opensearch 1.1 cluster. After selecting to upgrade it to 1.2 the cluster has been stuck on "Preparing to process updates". 
I found out that it is stuck because the default settings for .opendistro-* indices are to create 5 primaries and 1 replica shard. As our cluster only consists of a single shards those some of the .opendistro-* indices are not unallocated.
I created index templates for those indices in question with 1 primary and 0 replicas and deleted some of the .opendistro indices but there are .opendistro-alerting-alerts indices which I can not delete even though I have all the permissions possible.

Also followed this guide but still getting 403 https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-disable-index-alerts/

GET _cat/shards
.opendistro-alerting-alerts                            3 r UNASSIGNED                              
.opendistro-alerting-alerts                            1 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts                            1 r UNASSIGNED                              
.opendistro-alerting-alerts                            2 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts                            2 r UNASSIGNED                              
.opendistro-alerting-alerts                            4 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts                            4 r UNASSIGNED                              
.opendistro-alerting-alerts                            0 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alerts                            0 r UNASSIGNED     
.opendistro-alerting-alert-history-2022.03.16-1        3 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1        3 r UNASSIGNED                              
.opendistro-alerting-alert-history-2022.03.16-1        1 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1        1 r UNASSIGNED                              
.opendistro-alerting-alert-history-2022.03.16-1        2 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1        2 r UNASSIGNED                              
.opendistro-alerting-alert-history-2022.03.16-1        4 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1        4 r UNASSIGNED                              
.opendistro-alerting-alert-history-2022.03.16-1        0 p STARTED                   x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822
.opendistro-alerting-alert-history-2022.03.16-1        0 r UNASSIGNED

DELETE .opendistro-alerting-alert-history-2022.03.16-1
DELETE .opendistro-alerting-alerts
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]"
  },
  "status" : 403
}

I would gladly add some additional data nodes to the cluster in order to fix this but this also fails as the cluster is stuck on "Preparing to process updates".

EDIT:
I tried deleting the .opendistro indices with "m aster" user but even "m aster" user receives error 403.

AWS-User-7264420 · Answer

If we had premium support we wouldn't be asking for help in here.
I was under the impression that this is the place to come for help in case you don't have a paid support plan.

Could you answer why is the "\master" user not able to delete the indices I mentioned? If we'd get those deleted then we'd probably be able to carry on with the upgrade.

Also this site does not allow posting the word "\master".

Varun_S · Answer

When the cluster is in processing state, the cluster will try to complete the blue/green deployment change for a limited number of times before getting stuck. Once the cluster has failed the maximum number of retries possible, the cluster will remain in processing state until intervened by AWS internal team. Please reach out to the premium support with the ARN of the cluster to get the cluster back to the active state.

Other recommendation is to please follow the best practices for Amazon OpenSearch Service: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html

Blue/green deployment: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html

Opensearch upgrade stuck

Relevant content