OpenSearch Snapshot S3 repository "cannot delete test data"

Question

I'm getting the following error when creating a S3 snapshot repository.

```
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[] cannot delete test data at []"}],"type":"repository_verification_exception","reason":"[] cannot delete test data at []","caused_by":{"type":"i_o_exception","reason":"Failed to list objects for deletion","caused_by":{"type":"completion_exception","reason":"software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: , Extended Request ID: ) (SDK Attempt Count: 1)","caused_by":{"type":"s3_exception","reason":"Access Denied (Service: S3, Status Code: 403, Request ID: , Extended Request ID: ) (SDK Attempt Count: 1)"}}}},"status":500}
```

Although the repository is created and the test directories are created on S3 I'm get the error above. And as suggested by the error, the test directories on S3 aren't deleted.

Follows the snapshot IAM policy

```
{
    "Statement": [
        {
            "Action": "s3:ListBucket",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::"
        },
        {
            "Action": "sts:GetCallerIdentity",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "iam:PassRole",
            "Effect": "Allow",
            "Resource": "arn:aws:iam:::role/"
        },
        {
            "Action": "es:ESHttpPut",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:es:eu-west-2::domain//*",
                "arn:aws:es:eu-west-2::domain//*"
            ]
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::/*"
        }
    ],
    "Version": "2012-10-17"
}
```

I've tried allowing `s3:*` but still get the same error.

And follows the respective trusted entities
```
{
    "Version": "2012-10-17",
    "Statement": [
         {
            "Effect": "Allow",
            "Principal": {
                "Service": "es.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "opensearch.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

re:Post Agent · Answer

The error you're encountering when creating an S3 snapshot repository in OpenSearch is related to permissions issues. While your repository is being created and test directories are appearing in S3, the service is unable to delete these test files, resulting in the "Access Denied" error.

Based on your IAM policy and trust relationship configuration, here are some potential issues and solutions:

1. **S3 Bucket Policy**: Check if your S3 bucket has a bucket policy that might be restricting delete operations. Even though your IAM role has the necessary permissions, the bucket itself might have restrictions.

2. **S3 Bucket Ownership**: Ensure that the bucket ownership settings aren't causing conflicts. If Object Ownership is set to "Bucket owner preferred" or "Object writer", it might affect permissions.

3. **Fine-grained Access Control**: If you have fine-grained access control enabled on your OpenSearch domain, you need to ensure that:
- The user making the request has the appropriate role mappings in OpenSearch
- The role has permissions for `cluster:admin/repository/put` and other snapshot-related actions

4. **Authentication Method**: When registering a snapshot repository with fine-grained access control enabled, you need to make a signed request with AWS Signature Version 4 authentication. If you're using the Dev Tools console in OpenSearch Dashboards, you might be making an unsigned request as an "anonymous" user from AWS's perspective.

5. **Role Mapping**: Ensure that your IAM role is properly mapped to a role in OpenSearch that has snapshot permissions. In OpenSearch Dashboards, go to Security > Roles > manage_snapshots, then choose Mapped users > Manage mapping, and add the ARN of your IAM role under Backend roles.

6. **Trust Relationship**: Your trust relationship looks correct with both `es.amazonaws.com` and `opensearch.amazonaws.com` as trusted services.

Try using a tool like Postman to make properly signed requests when registering the repository, as this can help ensure proper authentication with your AWS credentials.  
**Sources**  
[OpenSearch snapshots repo registration error: User: anonymous is not authorized to perform: iam:PassRole | AWS re:Post](https://repost.aws/questions/QUinFcHTm1RqCaqZIBc7-WqA/opensearch-snapshots-repo-registration-error-user-anonymous-is-not-authorized-to-perform-iam-passrole)  
[OpenSearch error: no permissions for [cluster:admin/repository/put] and User [name=arn:aws-us-gov:iam::123456789012:user/opensearch-repo-user | AWS re:Post](https://repost.aws/questions/QU-ReRhJDdRqq4ZXf_OxvN4w/opensearch-error-no-permissions-for-cluster-admin-repository-put-and-user-name-arn-aws-us-gov-iam-123456789012-user-opensearch-repo-user)  
[Creating index snapshots in Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html)  
[Opensearch manual snapshot issues | AWS re:Post](https://repost.aws/questions/QUXlNBHa-7Spml03LBxT7Qlg/opensearch-manual-snapshot-issues)

Taz · Answer

That error means OpenSearch can write the temporary test file to your S3 bucket but can’t delete it afterward during repository verification. The snapshot repository plugin needs both write and delete access to the target bucket and its objects. In your IAM policy, you likely have `s3:PutObject` and `s3:ListBucket`, but either `s3:DeleteObject` or the correct bucket ARN is missing. The resource ARNs must point to the exact bucket and prefix, for example `arn:aws:s3:::my-snapshot-bucket` for `ListBucket` and `arn:aws:s3:::my-snapshot-bucket/*` for object-level actions.
—Taz

OpenSearch Snapshot S3 repository "cannot delete test data"

Add your answer

Relevant content