Export Elasticache backup to s3 using boto3

0

Export working from console, but when trying to export Elasticcache backup to s3 using boto3 (from lambda function) I am getting the following error:

[ERROR] InvalidParameterValueException: An error occurred (InvalidParameterValue) when calling the CopySnapshot operation: Elasticache was unable to validate the authenticated user has access on the S3 bucket BACKUP_NAME.

The bucket ACL configured correctly according to this guide

Python boto3:

ec_client = boto3.client('elasticache')
response = ec_client.copy_snapshot(
                SourceSnapshotName=source_snapshot_name,
                TargetSnapshotName=target_snapshot_name,
                TargetBucket=s3_bucket_name
            )

Lambda role's permissions:

{
    "Statement": [
        {
            "Action": [
                "elasticache:DescribeSnapshots",
                "elasticache:CopySnapshot"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketAcl",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Effect": "Allow",
            "Resource": [
                "BUCKET_NAME/*",
                "BUCKET_NAME"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Bucket ACL:

External account
Canonical ID:
540804c33a284a299d2547575ce1010f2312ef3da9b3a053c8bc45bf233e4353     Objects: List, Write  Bucket ACL: Read, Write
3 Answers
1
Accepted Answer

Apparently the copy_snapshot method in boto3 requires the permission to list all buckets. This issue was resolved by updating the Lambda role's permissions with the following statement:

{
    "Action": [
        "s3:ListAllMyBuckets"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::*"
    ]
}

This statement grants the Lambda role permission to list all buckets in Amazon S3.

Ron
answered 16 days ago
1

Same error even with s3:*. I believe the problem is related to ACL permissions because the same error occurred when I tried to export the snapshot via the console before I configured the bucket ACL as described in the documentation. But now everything is configured as it should. Maybe some kind of integration problem with lambda or boto3?

Cloudtrail access denied event:

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "PRINCIPAL_ID:ROLE_NAME",
        "arn": "arn:aws:sts::ACCOUNT_ID:assumed-role/ROLE_NAME/ROLE_NAME",
        "accountId": "ACCOUNT_ID",
        "accessKeyId": "ACCESS_KEY_ID",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "PRINCIPAL_ID",
                "arn": "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME",
                "accountId": "ACCOUNT_ID",
                "userName": "ROLE_NAME"
            },
            "attributes": {
                "creationDate": "2024-01-31T08:28:02Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "ec.amazonaws.com"
    },
    "eventTime": "2024-01-31T08:28:04Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "ListBuckets",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "ec.amazonaws.com",
    "userAgent": "ec.amazonaws.com",
    "errorCode": "AccessDenied",
    "errorMessage": "Access Denied",
    "requestParameters": {
        "Host": "s3.us-east-1.amazonaws.com"
    },
    "responseElements": null,
    "additionalEventData": {
        "SignatureVersion": "SigV4",
        "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "bytesTransferredIn": 0,
        "AuthenticationMethod": "AuthHeader",
        "x-amz-id-2": "x-amz-id-2",
        "bytesTransferredOut": 243
    },
    "requestID": "4AFDSGDSFGFSQKWN",
    "eventID": "3b64567-1da4-4c42-846c-80c35546ea14",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "ACCOUNT_ID",
    "vpcEndpointId": "VPC_ENDPOINT_ID",
    "eventCategory": "Management"
}
Ron
answered 21 days ago
  • Could you share your bucket policy? It is to confirm if everything is as it supposed to be, because cloudtrail logs says that elasticache is accessing S3 and it brought

  • My bucket policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "elasticache.amazonaws.com"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:ListBucket",
                    "s3:GetBucketAcl",
                    "s3:ListMultipartUploadParts",
                    "s3:ListBucketMultipartUploads"
                ],
                "Resource": [
                    "arn:aws:s3:::BUCKET_NAME",
                    "arn:aws:s3:::BUCKET_NAME/*"
                ]
            }
        ]
    }
    
1
AWS
vtjean
answered 22 days ago
  • Thank you for your response. Same error even with s3:*. I believe the problem is related to ACL permissions because the same error occurred when I tried to export the snapshot via the console before I configured the bucket ACL as described in the documentation.

    But now everything is configured as it should. Maybe some kind of integration problem with lambda or boto3?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions