OpenSearch Deployment Failing with 403 Forbidden Despite Correct IAM and Security Configuration

Question

OpenSearch Configuration:
Endpoint: vpc-open-search-*******.us-west-2.es.amazonaws.com:443
Security Group Inbound Rules:
HTTPS (Port 443) allowed from specific security groups within the same VPC.
IAM Role and Policy:
We created an IAM role open-search with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2::domain/open-search/*"
    }
  ]
}

The IAM role has full access to AWS services and has been attached correctly to the application.

Issue:
Despite the above setup, the application is unable to communicate with the OpenSearch endpoint and fails with a 403 Forbidden error. However, the same configuration works perfectly in another AWS account.
What We Have Tried:
Verified IAM role policies and ensured they match the working account setup.
Confirmed VPC and Security Group configurations allow inbound traffic on port 443.
Checked the access policy for the OpenSearch domain to ensure the IAM role is listed.
Tested direct access using curl to check for any network issues.
Verified IAM user access from the CLI, and the user can successfully access OpenSearch using the describe-domain command:
aws es describe-domain --domain-name qa-open-search
This confirms the IAM user has the necessary permissions and can interact with the domain.
Questions:
What additional configurations or settings could cause the 403 error in this scenario?
Are there specific differences in access control between accounts that might affect OpenSearch domain access?
Any guidance or insights would be greatly appreciated!

re:Post Agent · Answer

Based on the information provided, there are a few potential reasons for the 403 Forbidden error you're encountering with your OpenSearch deployment:

1. OpenSearch Service API permissions: Starting May 10, 2023, OpenSearch Service requires two additional IAM permissions for collection resources. These are 'aoss:APIAccessAll' for data plane access and 'aoss:DashboardsAccessAll' for OpenSearch Dashboards access from the browser. If these permissions are not added to your IAM role, it could result in a 403 error.

2. Request signing: Ensure that your requests to the OpenSearch domain are properly signed. OpenSearch Service requires requests to be signed using Sigv4 with an access key ID and a secret access key associated with an IAM principal. Alternatively, you can use AWS Security Token Service (AWS STS) to generate temporary security credentials for signing requests.

3. TLS version: OpenSearch Service requires TLS 1.2 or higher. Ensure that your client is using a compatible TLS version. You can configure the minimum required TLS version using the 'TLSSecurityPolicy' value in the domain endpoint options.

4. Fine-grained access control: If fine-grained access control is enabled on your OpenSearch domain, make sure that the IAM role has the necessary permissions at the index, document, or field level.

5. VPC access: If your OpenSearch domain is configured for VPC access, ensure that your application is running within the same VPC or has proper network connectivity to the VPC where the OpenSearch domain is deployed.

6. Domain access policy: Double-check the access policy for the OpenSearch domain. It should explicitly allow access to the IAM role you're using.

To troubleshoot further:

1. Review the CloudTrail logs for any denied actions related to OpenSearch.
2. Use the AWS CLI to test access to the OpenSearch APIs directly.
3. Verify that the IAM role is correctly assumed by your application.
4. Compare the full IAM policy, including any attached AWS managed policies, between the working and non-working accounts.

If the issue persists after checking these points, you may need to contact AWS support for further assistance, as there could be account-specific configurations or limitations that are not immediately apparent.  
**Sources**  
[Open-Source Search Engine – Amazon OpenSearch Service Security – Amazon Web Services](https://aws.amazon.com/opensearch-service/features/security/)  
[Infrastructure security in Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/infrastructure-security.html)  
[Identity and Access Management for Amazon OpenSearch Serverless - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html)

OpenSearch Deployment Failing with 403 Forbidden Despite Correct IAM and Security Configuration

Add your answer

Relevant content