Cannot access OpenSearch with curl from EC2

Question

I have an Amazon OpenSearch service and an EC2 instance, each running in the same VPC and each uses the same Security Group. 
I have key credentials issued by AWS for a user named OpenSearchUser. To avoid specifics, let’s say my Access Key is 1234, and my Secret Key is ABCD.

I try to curl, employing my keys:
curl -v -X GET "https://vpc-rampart-test-opensearch-nrq.........cgq.us-east-1.es.amazonaws.com/" -H "Content-Type: application/json" --user 1234:ABCD --aws-sigv4 aws:amz:us-east-1:es

It gives me this error:

{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=arn:aws:iam::5.....1:user/OpenSearchUser, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=arn:aws:iam::5.....1:user/OpenSearchUser, backend_roles=[], requestedTenant=null]"},"status":403}

Because it is a service, I am unable to edit the trust policy for the service directly to add my user specifically.

The trust policy for my OpenSearch Service domain rampart-test-opensearch appears to allow any user in AWS to access the service. Here is the policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:[OUR-ACCT-ID-HERE]:domain/rampart-test-opensearch/*"
    }
  ]
}

What must I do to grant access to the OpenSearch service for OpenSearchUser from my EC2?

Answer

Have you included the IAM user as a principal of the Open Search Domain Access Policy?
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-types-resource

From https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html#fgac-access-policies

> Domain access policy
The second security layer is the domain access policy. **After a request reaches a domain endpoint, the resource-based access policy allows or denies the request access to a given URI.** The access policy accepts or rejects requests at the "edge" of the domain, before they reach OpenSearch itself.
.

Answer

I have an EC2 instance, NiFi 3.2. On this EC2 instance I run Apache NiFi.
This EC2 instance has IAM Role rampart-nifi-ec2.
IAM Role rampart-nifi-ec2 is granted three Permissions policies:
.....AmazonOpenSearchServiceCognitoAccess, which I am not currently using for anything
.....AmazonOpenSearchServiceFullAccess
.....AmazonS3FullAccess

Policy AmazonOpenSearchServiceFullAccess associates with two entities:
.....User OpenSearchUser
.....Role rampart-nifi-ec2

I created user OpenSearchUser as a User in AWS.
I generated an Access key and a Secret key for it.

I did this because I wanted a key pair for a user to test my PUT and GET access to Amazon OpenSearch from my EC2. After that test my eventual use case is to submit the curls from ExecuteStreamCommand within my NiFi flows, and that appears to require the key pair.

Cannot access OpenSearch with curl from EC2

Relevant content