OpenSearch cluster stuck in “Applying changes” after instance type update

Hi Tamirbr. Thanks for your question. I hope this information helps.

A stuck delete index task can block configuration changes like instance type upgrades in Amazon OpenSearch Service. Here's how to diagnose and resolve it:

Understanding the Problem

When you initiate a configuration change (like an instance type change), OpenSearch Service performs a blue/green deployment. Before starting this deployment, it runs validation checks. A pending delete task can cause the validation to fail or the deployment to get stuck because:

1. The cluster needs to be in a stable state for blue/green deployment
2. Pending cluster tasks can interfere with shard migration
3. The delete operation might be waiting for resources that aren't available

Diagnostic Steps

1. Check Cluster Tasks

Use the OpenSearch API to view pending tasks:

GET /_cat/tasks?v&detailed

Or for more detail:

GET /_cluster/pending_tasks

This will show you what tasks are stuck and why.

2. Check Cluster Health

GET /_cluster/health?pretty

Look for:

• Red or yellow cluster status
• Unassigned shards
• Relocating shards

3. Check Configuration Change Status

In the AWS Console:

• Go to your OpenSearch domain
• Check Domain processing status and Configuration change status
• Look for validation errors or stuck stages

Resolution Options

Option 1: Cancel the Configuration Change (If Possible)

If your configuration change is in a cancellable state:

Console:

1. Go to OpenSearch Service console
2. Select your domain
3. If available, click Cancel under Configuration change status

Note: You can only cancel during certain stages (Pending, Initializing, Validating, Awaiting user inputs)

Option 2: Clear the Stuck Delete Task

A. Check for Index Write Blocks

Stuck delete operations often occur when indexes have write blocks:

GET /_cluster/settings?include_defaults=true&filter_path=**.read_only*

If you find read-only blocks, remove them:

PUT /_cluster/settings
{
  "persistent": {
    "cluster.blocks.read_only": false,
    "cluster.blocks.read_only_allow_delete": false
  }
}

B. Check for Shard Allocation Issues

GET /_cluster/allocation/explain

This shows why shards can't be allocated. Common issues:

• Insufficient disk space
• Shard routing disabled
• Node availability issues

C. Force Delete the Index (If Safe)

If the index is truly no longer needed and the delete is stuck:

DELETE /your-index-name

If that fails, try with query parameters:

DELETE /your-index-name?timeout=5m&master_timeout=5m

D. Check for Snapshot Operations

Stuck delete tasks can occur if a snapshot is in progress:

GET /_snapshot/_status

If a snapshot is stuck, you may need to delete it:

DELETE /_snapshot/repository-name/snapshot-name

Option 3: Resolve Underlying Cluster Issues

Based on the validation errors documentation, common blockers include:

1. Red Cluster Status (RedCluster)

   • Identify red indexes: GET /_cat/indices?v&health=red
   • Fix or delete red indexes before proceeding

2. Low Disk Space (InsufficientStorageCapacity)

   • Delete unused indexes
   • Increase EBS volume size first
   • Add more nodes

3. Shard Movement Blocked (ShardMovementBlocked)

   • Check for shard allocation filters:

     GET /_cluster/settings?include_defaults=true

   • Remove problematic allocation rules:

     PUT /_cluster/settings
     {
       "persistent": {
         "cluster.routing.allocation.exclude._name": null,
         "cluster.routing.allocation.require._name": null
       }
     }

4. Too Many Requests (TooManyRequests)

   • Reduce indexing/search load temporarily
   • Wait for cluster to stabilize

Option 4: Contact AWS Support

If none of the above work, you may need AWS Support to intervene. They can:

• Force-cancel stuck configuration changes
• Clear stuck cluster tasks
• Investigate backend issues

When to contact support:

• Configuration change stuck for > 24 hours
• Unable to cancel the change
• Cluster is in an inconsistent state
• You've tried all troubleshooting steps

Prevention Tips

1. Always check cluster health before configuration changes:

   GET /_cluster/health
   GET /_cat/pending_tasks

2. Use dry run validation before making changes:

   • In Console: Use "Dry run analysis" before saving changes
   • Via API: Set DryRun=true in UpdateDomainConfig

3. Avoid configuration changes during:

   • Active snapshot operations
   • Heavy indexing periods
   • When cluster status is yellow or red
   • When there are pending cluster tasks

4. Monitor these metrics:

   • ClusterStatus.red or ClusterStatus.yellow
   • Nodes count
   • FreeStorageSpace
   • JVMMemoryPressure

Quick Command Reference

# Check cluster health
GET /_cluster/health?pretty

# View pending tasks
GET /_cluster/pending_tasks

# View all tasks
GET /_cat/tasks?v&detailed

# Check index status
GET /_cat/indices?v

# Remove read-only blocks
PUT /_cluster/settings
{
  "persistent": {
    "cluster.blocks.read_only": false,
    "cluster.blocks.read_only_allow_delete": false
  }
}

# Check shard allocation
GET /_cluster/allocation/explain

# Force delete index
DELETE /index-name?timeout=5m&master_timeout=5m

References

• Making configuration changes in Amazon OpenSearch Service
• Troubleshooting Amazon OpenSearch Service
• Managing indexes in Amazon OpenSearch Service

---

TL;DR: Check cluster health and pending tasks via OpenSearch APIs. Remove any read-only blocks, resolve red cluster status, ensure sufficient disk space, and try force-deleting the stuck index. If the configuration change is cancellable, cancel it, fix the underlying issue, then retry. If stuck for >24 hours, contact AWS Support.