Upgrading ElasticCache Cluster from 6.0.0 to 6.2.0 lead to write failures for about five to ten mins with the error **READONLY You can't write against a read only replica**

0

Problem Upgrading Cluster from 6.0.0 to 6.2.0 lead to write failures for about five to ten mins with the error READONLY You can't write against a read only replica

After sometime the issue got auto-resolved and writes started working just fine.

I replicated this multiple times and the behaviour seems to be consistent. Why is this happening?

How can we mitigate these failures?

Redis Configuration:

  • Using replicatedConfig with MultiAZEnabled.
  • Have one primary and secondary node.
  • Redis Engine: 6.0.0
  • Using Java's redisson library.
  • Default Redisson Library Config.
NarenCf
asked 7 months ago165 views
1 Answer
0

Hello,

Thank you for using AWS ElastiCache service.

Kindly note that this error is seen when the client application makes request to set a key or write some data by connecting to the replica node's endpoint.

Also, it is worth to consider here that during Engine Upgrade, the Redis engine version is updated on each node in the cluster one at a time. The primary node is upgraded first, followed by the replica nodes. During this process, Elasticache triggers a failover to switch to new primary and updates the DNS record to point to new primary. However, if the client has cached the old nodes IP and connects to the old primary, you will face the error "READONLY You can't write against a read only replica" as the old primary is no longer accepting writes.

Usual tendency of client application is to resolve the primary endpoint once and cache the primary IP address locally. During the failover the IP address on primary endpoint changes to new primary node. If your client does not detect this change, it will still try to connect to old IP and this will result in prolonged downtime.

To ensure less interruption, you should have proper retry configuration in place at client side to detect failover and pick new IPs. Also, you can consider to reduce the DNS caching TTL values (if used).

AWS
Arti_S
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions