Elasticsearch 6.4 - R20211203-P3 update deleted all documents?

1

We have been using the original ElasticSearch for a few years, works great. Today it stopped working. When I logged into the AWS console and went to the OpenSearch page our domain had a status of Processing and at first I thought it was stuck but after several minutes it finished. It then did the whole update process again which took another several minutes. Now it is back to Active but it still isn't working.

I think the indices have been deleted? I have no documents any more. Is there a back up to restore from? under the Cluster configuration it says Snapshots - frequency - hourly, but I don't see anyway to restore it.

Under the error logs I see:

[2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times)
ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused];
	at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165)
	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643)
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316)
	at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153)
	at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:749)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	... 1 more
Caused by: java.net.ConnectException: Connection refused
	... 10 more
[2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more
Ron D
gefragt vor 2 Jahren365 Aufrufe
1 Antwort
0

Hi Ron, you can restore the latest SUCCESSful automated snapshot using: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore

However, snapshots are only retained for 14 days and so is the question age. Would recommend taking action immediately.

AWS
Ajay N
beantwortet vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen