OpenSearch domain stuck in "ApplyingChanges" state for 7+ hours - cannot cancel or apply new changes

Question

Hi,

My AWS OpenSearch domain has been stuck in "ApplyingChanges" / 
"PROCESSING" state for 7+ hours and I need help resolving it.

**Problem:**
- Domain processing status: Modifying
- Configuration change status: Applying changes (stuck)
- Change has not progressed since it started

**describe-domain-change-progress output:**
{
    "Status": "PROCESSING",
    "ConfigChangeStatus": "ApplyingChanges",
    "LastUpdated": "never changes"
}

**What I tried:**
1. cancel-domain-config-change → Error: "CancelDomainConfigChange 
   only allowed when domain config change status is ValidationFailed"
2. update-domain-config → Error: "A change/update is in progress. 
   Please wait for it to complete"
3. Terraform apply → Same error as above

**Questions:**
1. How long should ApplyingChanges take for 
   InternalUserDatabaseEnabled change?
2. Is there a way to force complete or reset a stuck change?
3. Will this resolve automatically?

**Setup:**
- Single node t3.small
- OpenSearch 3.1
- VPC deployment
- Change type: AdvancedSecurityOptions.InternalUserDatabaseEnabled

Any help appreciated!

re:Post Agent · Answer

I understand your frustration with the domain being stuck in the "ApplyingChanges" state. Here's what's happening and what you can expect:

**Why Your Domain is Stuck:**

Your OpenSearch Service domain enters the "Processing" state when it's in the middle of a configuration change. During this process, OpenSearch Service uses a blue/green deployment approach where it creates a new environment with a new set of applicable nodes. The domain can get stuck if:
- New data nodes fail to launch properly
- Shard migration to the new nodes is unsuccessful
- Validation checks have failed with errors

**Important Limitations:**

Unfortunately, you **cannot cancel an upgrade or configuration change** after you initiate it. The cancel operation only works when the domain config change status is "ValidationFailed" - which explains the error you're receiving. Once a change is in the "ApplyingChanges" state, you must wait for AWS to resolve any underlying issues before the process can complete.

**What You Should Do:**

1. **Monitor CloudWatch Metrics:** Check metrics like "FreeStorageSpace," "ClusterStatus," and "ClusterIndexWritesBlocked" to identify if there are resource constraints causing the delay.

2. **Check for Resource Issues:** Even on a single t3.small node, ensure you have sufficient disk space and that the node isn't experiencing memory pressure or other resource constraints.

3. **Wait for Resolution:** AWS will work to resolve the underlying issue automatically. The domain should eventually complete the change or fail with a specific error message.

4. **Contact AWS Support:** Given that it's been 7+ hours, I strongly recommend opening a support case with AWS. They can investigate the specific issue causing your domain to be stuck and potentially expedite resolution.

The timeline for configuration changes varies based on the specific change type and domain size, but 7+ hours is unusually long and warrants AWS Support involvement.  
**Sources**  
[Troubleshooting Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html)  
[Troubleshoot stuck or failed upgrades in OpenSearch Service | AWS re:Post](https://repost.aws/knowledge-center/opensearch-stuck-failed-upgrade)

OpenSearch domain stuck in "ApplyingChanges" state for 7+ hours - cannot cancel or apply new changes

Add your answer

Relevant content