We have been using the original ElasticSearch for a few years, works great. Today it stopped working. When I logged into the AWS console and went to the OpenSearch page our domain had a status of Processing and at first I thought it was stuck but after several minutes it finished. It then did the whole update process again which took another several minutes. Now it is back to Active but it still isn't working.
I think the indices have been deleted? I have no documents any more. Is there a back up to restore from? under the Cluster configuration it says Snapshots - frequency - hourly, but I don't see anyway to restore it.
Under the error logs I see:
[2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times)
ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused];
at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165)
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643)
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316)
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153)
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:749)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
... 1 more
Caused by: java.net.ConnectException: Connection refused
... 10 more
[2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more
