Upgrade ES 6.3 to 6.4 seems stuck

Question

We're testing the upgrade from ES version 6.3 to 6.4 on the ElasticSearch domain of our testing environment.  
  
This upgrade seems stuck however. Right now it's been over 15 hours since the process was started, and the "Upgrade History" tab still shows "Upgrade - 53.79% completed" (It has been this percentage for the last few hours). The Kibana plugin seems to be upgraded, but the ES itself doesn't. When opening kibana, it indicates that 'one of the nodes' is on a lower version of ES. When I go to the /_nodes endpoint (or the data instances list under 'instance health') it shows two nodes: one on the correct 6.4 version and one on the 6.3 version. The domain should however only consist of one t2.small instance...  
  
Recently we've upgraded the same domain from 6.2 to 6.3, which took less than an hour. Since that time, there was no big change in the data volume on the domain.  
  
Some Information about the domain:  
* 1 t2.small instance  
* 10GB EBS SSD  
* Service Software Release R20181023  
* 2,450,076 indexed documents  
  
Does anyone else have the same problem, or is it normal that a minor version upgrade of ES takes this much time on a small domain?

Answer

UPDATE: The instance remained stuck for a longer period, and eventually I got the message 'Upgrade failed'  
  
Afterwards, Amazon support staff had sent me an email, inidicating that the failure was due to a parse exception, more specifcally:  
  
+"The validation of the geo_shape field has been made robust in 6.4 version by elasticsearch commit :-  
https://github.com/elastic/elasticsearch/issues/31428 , causing the documents to have failed only after upgrade. To resolve this issue, you will need to fix the existing documents for the value of location.loc before initiating upgrade again."+  
  
So for me, this is solved

Upgrade ES 6.3 to 6.4 seems stuck

Relevant content