Skip to content

AWS OpenSearch Domain completely freeze - can't even delete it

0

I was using the Free Tier configuration for OpenSearch testing the tool, and at some point during the night, I started getting 504 errors when accessing the dashboard, which later changed to 502 errors. I also couldn’t use the API due to timeouts.

I couldn’t identify the cause of the issue. The only other AWS services interacting with OpenSearch are a Lambda function, which wasn’t triggered at the time, and an Elastic Beanstalk Flask app that no one was even interacting with. (This is a test environment)

I even tried to delete the domain, but the deletion process got stuck at 0% during the "initialization phase" and stayed like that for a good 45 minutes.

After deleting the domain, I created a new one, but the same issue occurred again today. This time, I just waited, and after about 30 minutes, it started working again.

One thing I noticed is that the "Searchable Documents" count keeps increasing, even though I’m not adding any new documents.

Searcheable Documents increasing. The period without count is the period when it froze Similar oddity for Deleted documents

Does anyone have a hint as to what might be happening? A misconfiguration maybe?

asked a year ago431 views

1 Answer
0

Based on the information provided, it seems you're experiencing some issues with your AWS OpenSearch domain that are causing it to freeze and become unresponsive. There are a few potential factors to consider:

  1. Resource limitations: Since you're using the Free Tier configuration, it's possible that you're hitting resource limits, which could cause the domain to become unresponsive. The Free Tier has limited resources, and if your data or query load exceeds these limits, it can lead to performance issues.

  2. Ongoing processes: The increasing "Searchable Documents" count, even when you're not adding new documents, suggests that there might be some background processes or indexing operations occurring. This could be causing strain on the system, especially if it's unexpected or unintended.

  3. Cluster status: It's possible that your cluster status has turned red, which can happen due to various reasons such as lack of disk space, shard allocation issues, or other configuration problems. A red cluster status can cause the domain to become unresponsive and difficult to manage or delete.

  4. Stuck upgrade or configuration change: Sometimes, if an upgrade process or configuration change gets stuck, it can prevent normal operations and even deletion of the domain.

To address these issues, you could try the following:

  1. Check your cluster health and status through the AWS console or API if possible.

  2. Monitor your resource usage and consider upgrading to a paid tier if you're consistently hitting limits.

  3. Investigate any unexpected indexing or document count increases. You might need to adjust your indexing settings or check for any unintended data ingestion.

  4. If you're unable to delete the domain due to it being stuck, you may need to contact AWS support. They can help override automated processes or fix issues preventing deletion.

  5. When creating a new domain, carefully review your configuration settings to ensure they're appropriate for your use case.

  6. Set up CloudWatch alarms to monitor key metrics and alert you before issues become critical.

If the problem persists, it would be advisable to open a support ticket with AWS, especially if you're unable to delete or manage the domain. They can provide more specific guidance based on the exact state of your OpenSearch domain and potentially resolve any backend issues that are causing the freezes.
Sources
Engine upgrade stucked, how to delete domain | AWS re:Post
Troubleshooting Amazon OpenSearch Service - Amazon OpenSearch Service
Deleting an Amazon CloudSearch Domain - Amazon CloudSearch

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.