AWS Opensearch Alerts issue after upgrade from 2.9 to 3.1 version

This error is related to a compatibility issue with alerts after upgrading your Amazon OpenSearch Service domain from version 2.9 to 3.1.

The "PreferenceBasedSearchNotAllowedException" indicates that there's a conflict between the alerting feature's search routing preferences and the stricter routing policies implemented in OpenSearch 3.1. This is likely due to changes in how search routing works in the newer version.

When you perform an OpenSearch version upgrade, Amazon OpenSearch Service runs validation checks to ensure your domain is eligible for the update. However, sometimes configuration incompatibilities like this can emerge after the upgrade is complete.

To resolve this issue, you have a few options:

1. Recreate your monitors with settings compatible with OpenSearch 3.1. This would involve:

• Taking note of your existing monitor configurations
• Creating new monitors with the same alert conditions
• Ensuring they use notification channels instead of older alerting destinations

2. Check if there are any pending configuration changes or service software updates that need to be applied to complete the upgrade process fully.

3. If you're using Auto-Tune, try enabling it if it's not already on, as it might help optimize your domain's performance and resolve compatibility issues.

4. If the issue persists, you may need to contact AWS Support, especially if you notice that the upgrade appears stuck or incomplete in the console UI.

Remember that starting with OpenSearch version 2.0, notification channels replaced alerting destinations, and in version 2.3 and later, existing destinations are automatically migrated to notification channels. Your upgrade from 2.9 to 3.1 might have affected this migration process.
Sources
Sending email notifications in Amazon OpenSearch | AWS re:Post
Making configuration changes in Amazon OpenSearch Service - Amazon OpenSearch Service
OpenSearch Upgrade Stuck at 60% – Urgent Assistance Required | AWS re:Post

I believe you have two options, depending on whether you want to disable this new shard routing enforcement or work around it.

Option 1: Disable strict weighted shard routing (Recommended for backward compatibility)

Run the following API call to update your cluster setting:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.use_strict_weighted_shard_preference": false
  }
}

Or via Dev Tools:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.use_strict_weighted_shard_preference": false
  }
}

This will restore support for preference-based searches, and your existing monitors will start working immediately without reconfiguration.

No data loss, and no monitor recreation needed.

Option 2: Manually update all monitors to remove preference-based routing (Not ideal)

You can technically:

Clone your existing monitors

Replace the search input with a custom query

Ensure the query does not use _search?preference=_some_preference_value

But:

This is time-consuming

Not always feasible if you have 100+ monitors

Not required if you can safely disable the setting above

AWS OpenSearch Service Note

If you're using Amazon OpenSearch Service (Managed):

Go to Cluster settings → Advanced cluster settings

Search for cluster.routing.use_strict_weighted_shard_preference

Set to false and Save

If you don’t have access (e.g., via Dev Tools/API), open a support ticket to update it from backend.

Final Recommendation(BASED ON YOUR ARCHITECTURE)

In production environments where you depend on Alerting and have legacy monitors:

Set cluster.routing.use_strict_weighted_shard_preference = false persistently.

This is the safest and AWS-recommended approach for users upgrading from 2.x to 3.x and relying on Alerting.

Manvitha, thanks for your response.

We tried switching cluster.routing.use_strict_weighted_shard_preference to false in Dev Tools, but received a response indicating that the request is not allowed. When we attempted the change via curl, we got the following error:

{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"}]}}

Additionally, this setting does not appear in the /_cluster/settings output.

This issue affects new monitors as well — any monitor created from scratch returns an empty result, while running the same Query DSL in the Discover tab returns the expected output.

Our queries do not use the _search API directly, so this does not seem to be related to search requests specifically.

In the AWS Console → OpenSearch → Edit cluster configuration → Advanced cluster settings, there is also no cluster.routing.use_strict_weighted_shard_preference option available. The only settings visible are Field data cache allocation and Max clause count.

Thanks, Yuriy

We encountered this issue as well after upgrading to OpenSearch 3.1. What’s happening is that 3.1 enables cluster.routing.use_strict_weighted_shard_preference by default, and the Alerting plugin’s preference-based searches don’t work with that turned on. On self-managed clusters, you could flip it off, but Amazon OpenSearch Service doesn’t expose this setting, which is why you don’t see it in /cluster/settings or the console.

That’s why Discover still works (it doesn’t use preference-based routing), but monitors break. Unfortunately, there’s no way to fix this yourself; the only option is to open an AWS Support case and ask them to disable cluster.routing.use_strict_weighted_shard_preference on your cluster. Once they do, your monitors (existing and new) should start working again.

you’ve done the right checks, but this one needs AWS Support to change it on the backend.

We have several AWS accounts with almost identical setups. All of them run the latest version of OpenSearch. However, this issue occurs only on the account where monitors and alerts are in use. We tried configuring the same alert on another AWS account, and the query worked correctly there. Additionally, we do not have AWS Support enabled on the account where the issue occurs, so we cannot ask about changing this setting.

-Yuriy