unable to update watermark limits in aws opensearch cluster. Got '/_cluster/settings' payload is not allowed as the error.

Hi Sai Teja,

In order to answer your query regarding the cluster settings, I would like to confirm that you received the following error message as you tried to perform an operation that is not supported in Amazon Opensearch cluster - {"Message":"Your request: '/_cluster/settings' payload is not allowed."}.

I would like to inform you that as Amazon Opensearch Service is a managed service, OpenSearch does not allow customers change the certain settings like watermark setting. To do so you can send request to AWS Support by creating a support ticket and requesting to change the watermark settings to desired bytes/ratio based on your use case. I would kindly request to refer below documentation to know more about the supported operations in Amazon OpenSearch Service.

[+] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html

OpenSearch Service only accepts PUT requests to the _cluster/settings API that use the "flat" settings form. It rejects requests that use the expanded settings form Example :

// Accepted
PUT _cluster/settings
{
  "persistent" : {
    "action.auto_create_index" : false
  }
}

// Rejected
PUT _cluster/settings
{
  "persistent": {
    "action": {
      "auto_create_index": false
    }
  }
}

The error you're encountering is due to the way AWS OpenSearch Service handles certain API requests differently from standard Elasticsearch or OpenSearch. To update cluster settings like watermark limits in AWS OpenSearch, you need to modify your request format.

Instead of using the structure you provided, try formatting your request like this:

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.low": "80%"
  }
}

Note that the key names are flattened and don't use nested objects. This format is specific to AWS OpenSearch Service and helps avoid the "payload not allowed" error.

If you're still encountering issues after reformatting your request, you may need to contact AWS support. Some cluster settings modifications might require intervention from the AWS OpenSearch team, especially if they involve critical system parameters.

Remember that changing watermark settings can significantly impact cluster behavior, so ensure you understand the implications before making these changes. It's also worth noting that in managed services like AWS OpenSearch, certain operations might be restricted for stability and performance reasons.
Sources
How do I implement semantic search with AWS OpenSearch service? | AWS re:Post