By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Indices from AWS Opensearch mysteriously disappear

0

Hello Everyone,

I have noticed that the indices are getting cleared from my Elasticsearch domain. I absolutely have no idea why they get deleted. I dont know which logs to look at. My domain settings:

  • Elasticsearch 7.10
  • 1-AZ without standby
  • t3.medium.search
  • Storage type: EBS
  • EBS volume type: General Purpose (SSD) - gp2
  • EBS volume size: 20 GiB
Topics
ServerlessAnalytics
Tags
Amazon OpenSearch Service
Language
English
Mohamed Sanaulla
asked a year ago984 views
1 Answer
0
Accepted Answer

Hello,

If your complete data is getting deleted, then it might be that you have single node cluster which is experiencing node drop hence causing data loss. As in case of a single node cluster only primary shards are allocated and there are no replica shards available for recovering data after node drop.

If this is the case you can check, SearchableDocuments metric (Maximum statistics with 1 min period) and compare it with Nodes metric (minimum statistics with 1 min period) in CloudWatch console. Keep both the metrics on different axis and see if the drop in SearchableDocuments metric is aligning with drop/missing data points in Nodes metric. This should help you confirm if node drop is the cause of data loss.

If above discussed scenario is not your issue and only few of your indices are getting deleted then I will suggest to check internally if DELETE requests are being sent from your end.

If you are not able troubleshoot the issue and require further assistance, please reach out to support engineering to check your domain and assist you further with the cause of data loss.

AWS
SUPPORT ENGINEER
Rajat_C
answered a year ago
  • Mohamed Sanaulla
    a year ago

    Thanks a lot @Rajat for your answer. This is exactly the reason why we faced a drop in indices. What should I do to create a replica node?

  • Rajat_C SUPPORT ENGINEER
    a year ago

    Hello Mohamed, Thank you for writing back - I am glad to hear that above information was helpful.

    In order to create a replica node, Please follow below steps:

    1. Add more data nodes in your domain. In OpenSearch service replica and primary shards cannot be assigned to same node hence we recommend to have atleast three nodes in domain to avoid potential OpenSearch issues, such as split brain and quorum loss.

    If you have cost constraints, having at-least two data nodes should be helpful for replica shards.

    [+] Sizing Amazon OpenSearch Service domains - Step 1: Make an initial estimate - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html#initial-estimate

    1. Once you have add more data nodes, go ahead and check if your indices have replica shards enabled or not. By default all indices have replica shards enabled. If you have custom shard configuration for your index, Please go ahead and check your index settings for sharding configuration.
    GET /<index-name>/_settings

    If replica shards are not enabled, update the index level setting of each index to enable replica shards.

    PUT /<index-name>/_settings
{
  "index" : {
    "number_of_replicas" : 1
  }
}

    You can also consider using Index template for new indices which are created, but for already available index you have to manually update the index settings (if replica shards are not enabled).

    [+] https://opensearch.org/docs/latest/im-plugin/index-templates/

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content