Amazon opensearch service keeps getting timed our after upgrade.

When your Amazon OpenSearch Service domain is experiencing timeouts after an upgrade, this could be related to several issues that commonly occur during or after configuration changes.

First, check if your domain is still in the "Processing" or "Modifying" state. After an upgrade, OpenSearch Service performs a blue/green deployment, creating a new environment and routing traffic to it. During this process, you might experience performance issues including increased latency and request rejections.

If your domain appears stuck in the processing state, you should:

1. Monitor CloudWatch metrics to identify potential performance bottlenecks
2. Check for any validation errors that might have occurred during the upgrade
3. Verify your configuration settings to ensure they're appropriate for your workload
4. Review logs for any specific errors related to the timeout issues

In some cases, domains can get stuck in the "Processing" or "Modifying" state. If this persists for an extended period, you may need to create a technical support case with AWS, especially if your data is important and you cannot recreate the cluster.

Some users have resolved similar issues by creating a new OpenSearch instance and migrating their data. However, this should be considered as a last resort if other troubleshooting steps fail.

If the 504 Gateway Timeout errors persist even after the domain shows as active, you might need to review your instance types and cluster configuration to ensure they can handle your workload after the upgrade.
Sources
Amazon OpenSearch Service Stuck at Modifying | AWS re:Post
AOSOPS04-BP01 Train staff on common OpenSearch issues and how to remediate them - Amazon OpenSearch Service Lens
OpenSearch Configuration Change Stuck at 20% and 503 Error Responses | AWS re:Post
Making configuration changes in Amazon OpenSearch Service - Amazon OpenSearch Service