NetworkPacketsPerSecondAllowanceExceeded is high

Question

When the *NetworkPacketsPerSecondAllowanceExceeded* metric is greater than 0, I calculated the sum of GetTypeCmds(4M, max at 13M) and SetTypeCmds(4M, max at 13M), which shows an average request rate of around 100,000 operations per second, with a peak reaching 430,000 operations per second at one point.

The read and write operations show a similar trend, indicating that the request volume for both is relatively balanced over time.

The current node type is cache.r7g.2xlarge, and memory utilization is only 2%, reaching up to 4% at peak. This indicates that memory is over-provisioned for our use case. The current issue appears to be related to the NetworkPacketsPerSecondAllowanceExceeded metric.

Cache Hit Rate - about 20%
CPU Utillization - about 70%, EngineCPUUtilization - about 50%
Bytes in - about from 3.5G to 4G, Bytes out - about from 15G to 20G

Since I'm currently using disabled cluster mode, I plan to add a read replica(with node cache.m7g.2xlarge) to see if it can help offload some of the network pressure.

Does anyone have any other suggestions regarding this change?

By the way, when I tested using the primary-replica setup, I noticed that the **SetTypeCmds metric on the replica node also had values**. As I understand it, the replica should only handle read operations and not perform any write operations. 
Is this because the replica is executing write commands during the replication process from the primary node?

Regards~

Refers: https://repost.aws/knowledge-center/elasticache-network-allowance-exceeded

re:Post Agent · Answer

When your NetworkPacketsPerSecondAllowanceExceeded metric is greater than 0, it indicates that your ElastiCache node is exceeding its packets per second (PPS) limit. This is happening despite your low memory utilization (only 2-4%), which confirms that memory isn't your bottleneck.

Based on your workload characteristics:
- High operation rates (100,000 ops/sec average, peaking at 430,000 ops/sec)
- Balanced read/write operations
- Moderate CPU utilization (70% overall, 50% engine)
- Low cache hit rate (20%)
- High network traffic (3.5-4G bytes in, 15-20G bytes out)

Your plan to add a read replica with a cache.m7g.2xlarge node may help with read operations, but since your workload appears to be balanced between reads and writes, it might not fully resolve the issue. For cluster mode disabled (CMD) clusters, there are some important considerations:

1. Read replicas will only help offload read operations, not writes, as all writes must go to the primary node.

2. Since your operations are balanced between reads and writes, you might still face PPS limitations on the primary node for write operations.

3. For write-heavy or balanced workloads in CMD clusters, scaling up to a larger node type is typically the recommended approach.

Additional suggestions:

1. Consider upgrading to a larger node type for your primary node to handle the high PPS.

2. Evaluate moving to cluster mode enabled (CME) which would allow you to distribute both reads and writes across multiple shards.

3. Review your application to see if you can batch operations to reduce the number of small requests, which would decrease your PPS.

4. Analyze your low cache hit rate (20%) - improving this could reduce the overall number of operations needed.

5. If possible, implement client-side caching to reduce the number of requests to ElastiCache.

Given your balanced read/write workload and CMD architecture, adding a read replica might provide some relief but likely won't completely resolve the PPS issue. A node type upgrade or migration to CME would be more effective for your specific use case.  
**Sources**  
[Troubleshoot Network allowance exceeded metrics in ElastiCache | AWS re:Post](https://repost.aws/knowledge-center/elasticache-network-allowance-exceeded)  
[Host-Level Metrics - Amazon MemoryDB](https://docs.aws.amazon.com/memorydb/latest/devguide/metrics.HostLevel.html)  
[Persistent connection issues - Amazon ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/TroubleshootingConnections.html)

NetworkPacketsPerSecondAllowanceExceeded is high

Add your answer

Relevant content