Opensearch ingestion pipeline from S3 in different account

• Make sure cross account access is setup correctly- https://repost.aws/knowledge-center/cross-account-access-s3.
• Please make sure correct action is specified "Action": "s3:", "Resource": "arn:aws:s3:::LOG_BUCKET_IN_ACCOUNT_B/". Also, for object level access (GetObject) correct ARN- "Resource": "arn:aws:s3:::LOG_BUCKET_IN_ACCOUNT_B/*".
• Check S3 Bucket Encryption- https://repost.aws/knowledge-center/cross-account-access-denied-error-s3

Yes I'm pretty sure that is setup correctly. The bucket uses default SSE-S3 encryption. I tested the cross account access with CLI and it worked fine.

Hi there,

Kindly see below the necessary policies/permissions/pipeline config etc. to set up a working cross-account pipeline from S3 and SQS to OpenSearch both with and without the use of KMS encryption on the S3 bucket:

WITHOUT KMS encryption on S3 bucket

In Account B --> pipeline and sink (OpenSearch domain)

pipeline config

version: "2"
s3-log-pipeline:
  source:
    s3:
      acknowledgments: true
      notification_type: "sqs"
      compression: "none"
      codec:
        newline:
      sqs:
        # Provide a SQS Queue URL to read from
        queue_url: "<SQS queue url>"
      aws:
        region: "<region>"
        sts_role_arn: "<pipeline IAM role arn>"
  sink:
    - opensearch:
        hosts: [ "<OpenSearch endpoint with https://>" ]
        aws:
          sts_role_arn: "<pipeline IAM role arn>"
          region: "<region>"
        index: "<index name>"

pipeline IAM role

Trust relationships > Trusted entities:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "osis-pipelines.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Permissions > Permissions policies:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OpenSearchaccess",
            "Effect": "Allow",
            "Action": [
                "es:DescribeDomain",
                "es:ESHttp*"
            ],
            "Resource": "<OpenSearch cluster arn>"
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "<S3 bucket arn>/*"
        },
        {
            "Sid": "ReceiveAndDeleteSqsMessages",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:ReceiveMessage",
                "sqs:receivemessage"
            ],
            "Resource": "<SQS queue arn>"
        }
    ]
}

If the OpenSearch cluster has fine-grained access control (FGAC) enabled, the following steps need to be taken:

FGAC OpenSearch cluster

Navigate to OpenSearch Dashboards > Security > Roles > "all_access" role > Mapped users > Manage mapping

Under Backend roles, add your pipeline IAM role's arn

Select Map

In Account A --> S3 bucket and SQS queue [and KMS key]

S3 bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<Account B ID>:root",
                    "arn:aws:iam::<Account A ID>:root",
                    "<pipeline IAM role arn>"
                ]
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "<S3 bucket arn>",
                "<S3 bucket arn>/*"
            ]
        }
    ]
}

SQS queue access policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage",
        "sqs:SendMessage"
      ],
      "Resource": "<SQS queue arn>",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": [
            "<Account A ID>",
            "<Account B ID>"
          ]
        },
        "StringLike": {
          "aws:SourceArn": "<S3 bucket arn>"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "<pipeline IAM role arn>"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<pipeline IAM role arn>"
      },
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage"
      ],
      "Resource": "<SQS queue arn>"
    }
  ]
}

WITH KMS encryption on S3 bucket

Please add the following KMS permissions to your pipeline IAM role in Account B:

KMS permissions

 {
            "Sid": "KMSKeyAccess",
            "Effect": "Allow",
            "Action": [
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:Decrypt"
            ],
            "Resource": [
                "<KMS key arn>"
            ],
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "s3.us-east-1.amazonaws.com"
                },
                "StringLike": {
                    "kms:EncryptionContext:aws:s3:arn": [
                        "<S3 bucket arn>/*",
                        "<S3 bucket arn>"
                    ]
                }
            }
}

Please add your pipeline IAM role arn to the "Allow use of the key" and "Allow attachment of persistent resources" sections of your KMS key policy in Account A:

KMS key policy

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            ...
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<pipeline IAM role arn>"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<pipeline IAM role arn>"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

Reference links:

[+] Amazon S3 as a source https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-s3.html#s3-source

[+] Providing cross-account ingestion access https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client.html#configure-client-cross-account

[+] SQS --> Example 4: Grant cross-account permissions to a role and a username https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-basic-examples-of-sqs-policies.html

[+] How do I configure my cross-account Amazon SQS endpoint to the Amazon SNS topic? https://repost.aws/knowledge-center/sns-and-sqs-cross-account-subscription

[+] How can I provide cross-account access to objects that are in Amazon S3 buckets? https://repost.aws/knowledge-center/cross-account-access-s3

[+] Step 1.3: Attach a bucket policy to grant cross-account permissions to Account B https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-walkthroughs-managing-access-example2.html#access-policies-walkthrough-example2a

[+] Delegating AWS permissions in a resource-based policy https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-cross-account-resource-access.html#access_policies-cross-account-delegating-resource-based-policies