Re-index in ElasticSearch with zero downtime
We use Elasticsearch 7.9, time to time we may need to do re-indexing. We are using aliases and the applications are pointing to alias instead of real index. I read some articles and it looks in most cases we would lose some update's/deletes that happen during the re-indexing process.
based on what I read, Below are the steps that I am planning to do.
Note : old_index is pointing to index-alias(it's alias name)
1- create new index(new_index) 2- do re-indexing from old_index to new_index 3- point the alias(index-alias) to new_index 4- Do second round of re-indexing from old_index to new_index
below are some of issues with this approach:
1- Two round of re-indexing needs to be done. This will become problematic, if we have large data in that index. 2- After step (3) , we may lose some of the deletes/updates on new_index.
What is the best method to do re-indexing with zero downtime and minimal impact for end-users?
Thanks in advance.
Does your data have any field to get the CDC like timestamp (for updated record) and id (for new record). If there is any field you can do the delta reindex on second iteration that will help speed up the second time reindex and will be way faster to sync the changes before you go live..
Migrating from RDS MariaDB 10.2 to Aurora MySQL 5.6Accepted Answerasked 3 years ago
What features of Open Distro is supported by ElasticSearch?Accepted AnswerMODERATORasked 3 years ago
Re-index in ElasticSearch with zero downtimeasked 3 months ago
Export data from AWS ElasticSearch to AWS RDS instance using Logstashasked 5 months ago
Logging with CloudWatch vs. ElasticSearch/KibanaAccepted Answerasked 5 years ago
Cognito & IAM setup for unauthorised access, to be used in Unity for Pollyasked a month ago
Is there a way to find where a private IP/domain points to?Accepted Answerasked 3 years ago
pagination question on appsync to elasticsearchAccepted Answerasked 3 years ago
Elastic Search pricing: Pay as you go or not?asked a year ago
ECS containers running on private subnet can not reach elastic searchAccepted Answerasked 3 years ago