Domain stuck in "Processing"

Hello,

If the cluster is stuck in Processing state, It would be suggested to raise a Support case AWS ES so that they find and fix if there is any workflow that is stuck and is leading to the cluster being in Processing state.

Looks like theres an open issue with ES domains not being able to complete loading or processing. We don't need to pay for technical assistance in order to report bugs so that AWS team can fix them.

Even while stuck in "processing" I was able to schedule the update to patch release R20200302. After this update was done my domain is now in "Active" state again. Go figure.

But I agree with foxy. It would be nice I one could raise issues with the ElasticSearch Service team when this happens. Because I feel it's a problem of the service and customers have no way of fixing these issues themselves. Unlike EC2 where you could for instance just launch a new instances when a instance fails.

The above gets even more annoying if you're under Basic Support and can't do anything about the Processing state. I'll open a new thread on that.

Hi ,

I have de same issue , i change my ES node instance type and the cluster status is stuck in "processing" por 24 hour and i see that not deploy blue/green occur becouse de count nodes is the same .

What can I do to fix this.

Region: us-east-1
Domain Name: plink-transactions

Hi ,

I have de same issue , i change my ES node instance type and the cluster status is stuck in "processing" por 24 hour and i see that not deploy blue/green occur becouse de count nodes is the same .

What can I do to fix this.

Region: us-east-1
Domain Name: plink-transactions

Still broken....

it's an absolute disgrace that AWS expect YOU to pay to notify them of THEIR bugs. Fix your broken service then I'll consider paying for support.

Edited by: JoshFreemanViion on Feb 1, 2021 4:21 AM

Today, AWS development team confirmed increased error rates impacting Create and Modify operations for Elasticsearch Domains in us-east-1 region but their dashboard doesn't report this.

Logging this one here.

Seeing domains stuck in "Loading" in us-east-1 and ca-central-1 all day today.
Domains are taking an unusually long time to delete.
Creating new domains does not work at all.

Create and Modify operations were experiencing increased error rates and latencies.
The issue has been resolved and the service is now operating normally.

Resolved in 15.5 hrs

Hi,

I got this problem yesterday (2021-04-06) because I added the EBS storage size to max limit and it's still in processing status after more than 24 hours. if you go to edit domain, the status of processing is in storage configuration (Storage configuration is Processing), data node is not in the list of instances and the status of the cluster is red. my account is basic how can i fix the problem

region: us-east-1
ES version: 7.4

Edited by: indraaarmy on Apr 6, 2021 7:55 PM

Hi,

My Domain Stuck in Processing status again, but health status is green.

region: us-east-1
ES version: 7.4

Edited by: indraaarmy on May 21, 2021 7:41 AM

same issue here, AWS expects you to buy a support plan to troubleshoot such issues which is totally unacceptable. my cluster is stuck in pending state form past 48 hours without support subscription i don''t know how am i going to resolve this issue?

There are several reasons why a cluster can be stuck in a processing state and none of them is due to a bug. This issue almost always affects clusters which do not follow the Elasticsearch Best Practice and the most common culprit is lack of resources due to too much load on the cluster.
This load can be due to:

a. Making configuration changes when the cluster is too busy. A cluster can be busy because of client requests or just because it is constantly relocating shards.
b. There are too many shards in the cluster given the available CPU cores and heap in your chose instance type. The workflow can timeout if there are too many shards to migrate. Ideally a node should have less than 1000 shards or have up 25 shards per GB of heap. The same can happen if the shards are too big.
c. Making concurrent configuration changes. AWS Elasticsearch uses other services like KMS for encryption, S3 for snapshots, IAM for permissions, etc. All these services need to know which is the current configuration version of Elasticsearch to point to. Making many changes at once with a poorly resourced cluster is bound to break the workflow somehow.

These are the major causes of broken workflows. Not a bug.

That is a load of rubbish @mpums

The AWS ElasticSearch is an absolute joke, it's utter useless. Want to restart it? No chance. Want to know why it's stuck processing? Go through 32 different trial and error and you'll find out it's none of them.

I had an ES that was running fine, its literally there just to be connected to and doesn't hold any data. It's practically 0% Storage usage, it's 1-2% cpu/memory usage, it has 2 shards, it's private, it gets about 3 connections a day and for some f*cking random ass reason it crashed 4 days ago. FINE for weeks, then poof random node crash.... When it's basically sat idle..

I've since moved everything to a custom EC2 because a monkey can run ES better than AWS can.

Same thing. Completely down an out right now because of this