Locking Against AWS MemoryDB Cluster Fails with: ERR WAIT cannot be used with replica instances

Question

We are using Redisson and AWS MemoryDB cluster for application locking. We are running into intermittent failures during locking operations. The error from Redis is:

`ERR WAIT cannot be used with replica instances. Please also note that since Redis 4.0 if a replica is configured to be writable (which is not the default) writes to replicas are just local and are not propagated.. channel: [id: 0xee5d2134, L:/172.30.80.188:52884 - R:10.51.142.62/10.51.142.62:6379] command: (WAIT), promise: java.util.concurrent.CompletableFuture@2a48dc53[Not completed], params: [1, 1000]`

We are configuring the Redisson client programmatically:

`org.redisson.config.Config config = new org.redisson.config.Config();
   config.useClusterServers()
               .addNodeAddress(redisUrl)   
               .setUsername(user)
               .setPassword(password)
               .setRetryAttempts(5)
               .setRetryInterval(5000)
               .setTimeout(10000);`

the redisUrl is a typical MemoryDB cluster URL specified by AWS that looks like this:

`clustercfg..0cfvkh.memorydb.us-east-1.amazonaws.com:6379 `

Reddison developers suggested this may be a MemoryDB issue where a host in the cluster redirects to itself.  See our ticket with Redisson, including detailed logs of application failures:

https://github.com/redisson/redisson/issues/4573

Answer

Hello,

Thank you for reaching out to us.

As per error messages in log file, it seems the issue is due to MOVED error received while connecting to cluster.

```
"reply: -MOVED 14005 memory-db-*********.amazonaws.com:6379  
-ERR WAIT cannot be used with replica instances.

"reply: -MOVED 7319 memory-db-*********.amazonaws.com:6379  
-ERR WAIT cannot be used with replica instances.
```

You received the error message "ERR WAIT cannot be used with replica instances " primarily due to moved reply received by your client Redisson.

The MOVED redirection is a normal response from the Redis nodes when they receive requests for hash slots they don't serve. It can also occur in other scenarios such as reading from replica node without specifying ReadOnly parameter .

[+] https://redis.io/commands/readonly/

[+] https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-redis-client-readonly/

It doesn't indicate an issue on the cluster side. The client should be able to handle this redirection and connect to the correct node for the requested key.

In Redis with cluster-mode-enabled, the keys are partitioned across the available shard. Each shard is responsible for serving a specific set of hash slots. The Redis client can query the cluster topology to determine which nodes are responsible for which slots. The MOVED is a redirection message that occurs when a client tries to access a key that belongs to a hash slot that is not served by the node the client is connected to. The response of the MOVED message contains the hash slot for the requested key, and Port of the node responsible for this hash slot.

In your case hash slot for the key to be retrieved is 14005 and 7319 respectively. This can be verified by running the CLUSTER SLOTS command.

To overcome this, the client should be configured to query the cluster topology by issuing CLUSTER SLOTS  or CLUSTER NODES commands frequently to get the latest mapping.

[+] https://redis.io/commands/cluster-nodes/

[+] https://redis.io/commands/cluster-slots/

You can also check the referenced hash slots by visiting "Shards and nodes" tab under your Memory DB cluster details.

I also found some articles about data paritioning and sharding from Reddison which may be of help to you.

[+] https://github.com/redisson/redisson/wiki/5.-data-partitioning-(sharding)

[+] https://github.com/redisson/redisson/wiki/7.-distributed-collections/#711-map-eviction-local-cache-and-data-partitioning

I hope the information above would help. Please feel free to reach out in case you have any additional queries

Locking Against AWS MemoryDB Cluster Fails with: ERR WAIT cannot be used with replica instances

Relevant content