How do I troubleshoot a “No endpoints available” error in the DynamoDB Accelerator (DAX) client?

Short description

"No endpoints available" errors occur when the client doesn't have a network route to any nodes of the DAX cluster. Or, these errors might occur when a cluster is down or facing high CPU utilization.

Here are some example cases that might result in one of these errors:

• A failed health check to DAX nodes: The client can't health check cluster nodes because of high CPU load from the nodes.
• Multiple consecutive IO Exceptions to each node: The DAX client counts the number of IO Exceptions to determine whether a server node is down. First, the client retrieves a list of server nodes from the configuration endpoint. If it receives more than five failures from a server node, then it’s removed the node from the list of active server nodes. If all server nodes are removed in this way, then the client ends up with an empty list of active nodes. In response, the client shows a "No endpoints available" error.

Resolution

To resolve your error and allow your cluster nodes to work normally, follow the troubleshooting steps below:

• In the DAX client, there’s a thread to refresh the list of active server nodes. Use the refresh option to restore your active nodes in the list.
• Increase the RequestTimeout based on the client’s traffic pattern. For more information, see Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications.
• Increase the health check interval. By default, health-check-interval is 5 seconds.
  Note: Increasing this interval puts the endpoint back into the client-side endpoint set only when it’s removed due to a socket-timeout-exception.
• To reduce the chance that these errors reoccur, scale up to a larger instance type. Or, scale out by adding more nodes to the cluster.

Related information

ClientConfig (DAX)