DAX errors in log - NoRouteException: not able to resolve address

0

Hello,

I created single node DAX cluster for testing purpose with R instance. In app logs I'm occasionally see below error for DAX cluster. Could you explain why it happens? There is already some discussion https://stackoverflow.com/questions/48135504/intermittent-dynamodb-dax-errors-norouteexception-during-cluster-refresh As I understand, it happens due to DAX cluster refresh which occurs on regular basis. Does this error affect the applications that rely on DAX? A deeper explanation would be appreciated.

at /usr/src/app/node_modules/amazon-dax-client/generated-src/ at SocketTubePool.alloc (/usr/src/app/node_modules/amazon-dax-client/src/Tube.js:263:64) Failed to pull from dax.yyy.dax-clusters.us-east-1.amazonaws.com (10.10.10.10): TimeoutError: ConnectionException: Connection timeout after 10000ms

at /usr/src/app/node_modules/amazon-dax-client/src/Source.js:123:14 at /usr/src/app/node_modules/amazon-dax-client/src/Source.js:86:20 at AutoconfSource._resolveAddr (/usr/src/app/node_modules/amazon-dax-client/src/Source.js:62:23)

Error: NoRouteException: not able to resolve address: [{"host":"dax.yyy.dax-clusters.us-east-1.amazonaws.com","port":9111,"scheme":"daxs"}] at /usr/src/app/node_modules/amazon-dax-client/src/Source.js:123:14 at /usr/src/app/node_modules/amazon-dax-client/generated-src/Operations.js:215:30 { at SocketTubePool.alloc (/usr/src/app/node_modules/amazon-dax-client/src/Tube.js:263:64) Failed to pull from dax.yyy.dax-clusters.us-east-1.amazonaws.com (10.10.10.10): TimeoutError: ConnectionException: Connection timeout after 10000ms

Thanks, OK

OK
asked 7 months ago282 views
1 Answer
0

Hello OK,

To answer your question - Does this error affect the applications that rely on DAX? The short answer is yes. The details of the error are shared in the knowledge center article

This error is more likely to occur with single node clusters. The 'NoRouteException: not able to resolve address' error is typically encountered when a client lacks a network route to any nodes within the DAX (DynamoDB Accelerator) cluster. This error may also manifest when the DAX cluster is either offline or under substantial CPU utilization. Several specific scenarios can lead to this error, including:

Failed Health Checks to DAX Nodes: In cases where DAX nodes experience high CPU loads, clients may fail to perform health checks on cluster nodes. This could result in an inability to connect to these nodes.

Multiple Consecutive I/O Exceptions to Each Node: DAX clients monitor I/O Exceptions to assess the operational status of server nodes. The client retrieves a list of server nodes from the configuration endpoint and registers any node that experiences more than five consecutive failures as inactive. When all server nodes become inactive in this manner, the client is left with an empty list of active nodes, triggering the 'NoRouteException: not able to resolve address' error.

*For production usage, we strongly recommend using DAX with at least three nodes, where each node is placed in different Availability Zones. Three nodes are required for a DAX cluster to be fault-tolerant. A DAX cluster can be deployed with one or two nodes for development or test workloads. One- and two-node clusters are not fault-tolerant, and we don't recommend using fewer than three nodes for production use. If a one- or two-node cluster encounters software or hardware errors, the cluster can become unavailable or lose cached data. * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.concepts.cluster.html

AWS
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions