How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?
3 minute read
My Amazon OpenSearch Service cluster has high JVM memory pressure, and I don't know how to resolve this.
By default, OpenSearch Service uses 50% of an instance's RAM for JVM heaps up to 32 GiB in size. The JVM memory pressure specifies the percentage of the Java heap in a cluster node. The following guidelines indicate what the JVM memory pressure percentages mean:
If JVM memory pressure reaches 75%, then OpenSearch Service initiates the Concurrent Mark Sweep (CMS) garbage collector for x86 Amazon Elastic Compute Cloud (Amazon EC2) instance types. ARM-based Graviton Amazon EC2 instance types use the Garbage-First (G1) garbage collector that uses additional short pauses and heap defragmentation. The garbage collection is a CPU-intensive process. If memory usage continues to grow, then you might encounter ClusterBlockException, JVM OutOfMemoryError, or other cluster performance issues. For more information, see Recovering from a continuous heavy processing load.
If JVM memory pressure exceeds 92% for 30 minutes, then OpenSearch Service blocks all write operations.
If JVM memory pressure reaches 100%, then OpenSearch Service JVM is configured to exit and eventually restarts on OutOfMemory (OOM).
The following reasons can cause high JVM memory pressure:
Spikes in the numbers of requests to the cluster.
Aggregations, wildcards, and selecting wide time ranges in the queries.
Unbalanced shard allocations across nodes or too many shards in a cluster.
Field data or index mapping explosions.
Instance types that can't handle incoming loads.
Reduce traffic to the cluster to resolve high JVM memory pressure issues. To reduce traffic to the cluster, follow these best practices:
Clear the field data cache with the POST /index_name/_cache/clear?fielddata=true API operation. Note: Clearing the cache can disrupt queries that are in progress.
Don't aggregate text fields or change the mapping type to "keyword".
Scale the domain so that the maximum heap size per node is 32 GB.
Delete old or unused indices to reduce the number of shards.
For advanced users, you can update the parent field data cache allocation, or request circuit breaker settings according to your use case. For more information about JVM circuit breakers, see JVM OutOfMemoryError.