What does opensearch error type: cluster_block_exception reason: blocked by: [FORBIDDEN/8/index write (api)....... mean?

Hello,

You might receive a ClusterBlockException error for the following reasons:

1. Lack of available storage space
2. High JVM memory pressure

[+] Troubleshooting Amazon OpenSearch Service - ClusterBlockException - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block

Lack of available storage space

If one or more nodes in your cluster has storage space less than the minimum value of 1) 20% of available storage space, or 2) 20 GB of storage space, basic write operations like adding documents and creating indexes can start to fail.

To avoid issues, monitor the FreeStorageSpace metric in the OpenSearch Service console and create CloudWatch alarms to trigger when FreeStorageSpace drops below a certain threshold. GET /_cat/allocation?v also provides a useful summary of shard allocation and disk usage. To resolve issues associated with a lack of storage space, scale your OpenSearch Service domain to use larger instance types, more instances, or more EBS-based storage.

High JVM memory pressure

When the JVMMemoryPressure metric exceeds 92% for 30 minutes, OpenSearch Service triggers a protection mechanism and blocks all write operations to prevent the cluster from reaching red status. When the protection is on, write operations fail with a ClusterBlockException error, new indexes can't be created, and the IndexCreateBlockException error is thrown.

When the JVMMemoryPressure metric returns to 88% or lower for five minutes, the protection is disabled, and write operations to the cluster are unblocked.

High JVM memory pressure can be caused by spikes in the number of requests to the cluster, unbalanced shard allocations across nodes, too many shards in a cluster, field data or index mapping explosions, or instance types that can't handle incoming loads. It can also be caused by using aggregations, wildcards, or wide time ranges in queries.

To reduce traffic to the cluster and resolve high JVM memory pressure issues, try one or more of the following:

• Scale the domain so that the maximum heap size per node is 32 GB.
• Reduce the number of shards by deleting old or unused indexes.
• Clear the data cache with the POST index-name/_cache/clear?fielddata=true API operation. Note that clearing the cache can disrupt in-progress queries.

In general, to avoid high JVM memory pressure in the future, follow these best practices:

• Avoid aggregating on text fields, or change the mapping type for your indexes to keyword.
• Optimize search and indexing requests by choosing the correct number of shards.
• Set up Index State Management (ISM) policies to regularly remove unused indexes.

[+] https://repost.aws/knowledge-center/opensearch-high-jvm-memory-pressure