Managed OpenSearch Cluster Error on Task Details

Question

We are experiencing critical performance degradation in our production OpenSearch cluster with unexplained high CPU usage across coordinator and data nodes, accompanied by ConcurrentModificationException errors in task management, despite no significant change in search request rates when we compare for last 2 weeks.

```
    {
      "task_id": 419572279,
      "node_id": "FM....3w",
      "status": "INTERNAL_SERVER_ERROR",
      "reason": {
        "type": "concurrent_modification_exception",
        "reason": "concurrent_modification_exception: null"
      }
    },
```

We have more than 40 data node and 3 coordinator and 3 master node. What does it mean this error exactly?

Kidd Ip · Answer

The concurrent_modification_exception in OpenSearch indicates that two or more threads attempted to modify the same internal data structure simultaneously, resulting in a race condition. In practice, this is not caused by your queries directly but by a bug or a concurrency issue in OpenSearch task management or query execution code. It manifests as INTERNAL_SERVER_ERROR in task details and can cause high CPU usage because the cluster repeatedly retries or fails tasks.

EXPERT

Kidd Ip

answered a year ago

Managed OpenSearch Cluster Error on Task Details

Add your answer

Relevant content