Elasticsearch 6.4 - R20211203-P3 update deleted all documents?

1

We have been using the original ElasticSearch for a few years, works great. Today it stopped working. When I logged into the AWS console and went to the OpenSearch page our domain had a status of Processing and at first I thought it was stuck but after several minutes it finished. It then did the whole update process again which took another several minutes. Now it is back to Active but it still isn't working.

I think the indices have been deleted? I have no documents any more. Is there a back up to restore from? under the Cluster configuration it says Snapshots - frequency - hourly, but I don't see anyway to restore it.

Under the error logs I see:

[2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times)
ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused];
	at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165)
	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643)
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316)
	at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153)
	at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:749)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	... 1 more
Caused by: java.net.ConnectException: Connection refused
	... 10 more
[2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more
Ron D
asked 2 years ago358 views
1 Answer
0

Hi Ron, you can restore the latest SUCCESSful automated snapshot using: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore

However, snapshots are only retained for 14 days and so is the question age. Would recommend taking action immediately.

AWS
Ajay N
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions