How do I troubleshoot slow logs in OpenSearch Service?

Resolution

I receive an error when I try to set up slow logs

If your AWS account exceeds ten resource policies in a single AWS Region, then you receive the following error in CloudWatch Logs: "Unable to create the Resource Access Policy - You have reached the maximum limit for number of Resource Access Policies for CloudWatch Logs. Please select an existing policy and edit it or delete an older policy and try again."

To resolve this error, create a resource policy that includes multiple log groups:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": [
        "logs:PutLogEvents",
        "logs:CreateLogStream"
      ],
      "Resource": [
        "ARN-Log-Group-1",
        "ARN-Log-Group-2",
        "ARN-Log-Group-3",
        "ARN-Log-Group-4"
      ]
    }
  ]
}

Note: You can't increase the AWS Identity and Access Management (IAM) policy quota.

I don't see any slow logs being delivered

If you don't see your slow logs delivered to CloudWatch, then check your IAM policy or OpenSearch Service thresholds.

Because OpenSearch Service requires permission to write to CloudWatch Logs, you must configure your IAM policy to log your queries.

To update your IAM policy, complete the following steps:

1. Open Search Slow Logs, and then choose Select Setup.

2. Update your IAM policy with your resource values:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "es.amazonaws.com"
         },
         "Action": [
           "logs:PutLogEvents",
           "logs:CreateLogStream"
         ],
         "Resource": "arn:aws:logs:us-east-1:588671893395:log-group:/aws/aes/domains/myes/search-logs:*"
       }
     ]
   }

3. (Optional) For additional security, add the aws:SourceAccount and aws:SourceArn condition keys to your policy:

   "Condition": {
     "StringEquals": {
       "aws:SourceAccount": "account-id"
     },
     "ArnLike": {
       "aws:SourceArn": "arn:aws:es:region:account-id:domain/domain-name"
     }
   }

   Note: Replace account-id with your account and domain-name with your domain name. For more information, see Turning on log publishing (console).

Set an appropriate timing threshold for your domain. If all your requests complete before the set threshold, then your logs won't be delivered to your log group. You can set individual index-level thresholds for each debug level (TRACE, DEBUG, INFO, and WARN).

For example, set the threshold for WARN debug levels to ten seconds for your index in OpenSearch Dashboards:

PUT /YOURINDEXNAME/_settings{"index.search.slowlog.threshold.query.warn": "10s"}

Note: Replace YOURINDEXNAME with your index name. You can set TRACE to "0" milliseconds to log all the queries for your domain. However, logging all queries is resource-intensive and might affect your domain performance.

To check your threshold, run the following command:

GET/YOURINDEXNAME/_settings?pretty

Note: Replace YOURINDEXNAME with your index name.

OpenSearch Service logs any queries that exceed the defined threshold.

Best practices

Take the following actions:

• Make configuration updates one at a time. Too many configuration changes at one time launch multiple blue/green deployments. This might cause the OpenSearch Service domain to get stuck in processing. For more information about blue/green deployment, see Making configuration changes in Amazon OpenSearch Service.
• To identify slow search queries, set a threshold for both the query phase and the fetch phase.
• Test with a low threshold value. Slowly increase the threshold to log only the queries that affect performance or require optimization.
• Choose the right number of shards for your cluster and use case to optimize cluster performance. For more information about shard maintenance, see Shard strategy.
• For slow logs, turn on logging at the TRACE, DEBUG, INFO, and WARN debug levels. Each debug level logs different categories of information. It's a best practice to turn on logging according to the request status.

Related information

Analyzing Amazon OpenSearch Service slow logs using Amazon CloudWatch Logs streaming and Kibana

How do I troubleshoot CloudWatch Logs so that it streams to my OpenSearch Service domain?

Viewing Amazon OpenSearch Service error logs