CloudFormation support for S3 replication to multiple destination buckets

0

As per https://aws.amazon.com/blogs/aws/new-amazon-s3-replication-adds-support-for-multiple-destination-buckets/, S3 now supports replication to multiple destination buckets, and according to the press release, it should be supported in CloudFormation.

However when adding the following configuration to CloudFormation:

PublicBucket:
    Type: AWS::S3::Bucket
    DependsOn: GlobalBuckets
    Properties:
      BucketName: !Sub learning-${Environment}-${AWS::Region}-public-web${Suffix}
      VersioningConfiguration:
        Status: Enabled
      ReplicationConfiguration:
        Role: !GetAtt PublicBucketReplicationRole.Arn
        Rules:
          - Id: ap-southeast-1
            Status: Enabled
            Destination:
              Bucket: !Sub arn:aws:s3:::learning-${Environment}-ap-southeast-1-public-web
          - Id: ap-southeast-2
            Status: Enabled
            Destination:
              Bucket: !Sub arn:aws:s3:::learning-${Environment}-ap-southeast-2-public-web

The deployment fails with the following error:

Number of distinct destination bucket ARNs cannot exceed 1 (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest; Request ID: EA29054861FE2AD9; S3 Extended Request ID: lbdTf_mHpoDLdCKp0w_bh38gjfcCKNF2Z7PmoIS/C6aMYGfdi1o8N1MS/MReNTRseuDPbo2y6LU=; Proxy: null)

The configuration works if I limit to a single replication rule.

Looks like this is actually NOT yet supported in CloudFormation?

asked 3 years ago1833 views
1 Answer
1

Found the solution - it is supported as of now, but not well documented.

Basically you need to ensure you force rules to use the new Replication Rules V2 schema to support multiple destination buckets. As per https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html#replication-backward-compat-considerations V2 schema is forced by specifying the Filter property on each rule.

Once you do this you need to ensure you add a number of configuration properties for each rule as per the example below, and you also need to ensure each Priority is a unique value.

Here is a working example:

Resources:
  PublicBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: foo-sandbox-us-west-2-public-web
      VersioningConfiguration:
        Status: Enabled
      ReplicationConfiguration:
        Role: !GetAtt PublicBucketReplicationRole.Arn
        Rules:
          - Id: ap-southeast-1
            Status: Enabled
            DeleteMarkerReplication:
              Status: Enabled
            Priority: 1
            Filter:
              Prefix: ''
            Destination:
              Bucket: arn:aws:s3:::foo-sandbox-ap-southeast-1-public-web
          - Id: ap-southeast-2
            Status: Enabled
            DeleteMarkerReplication:
              Status: Enabled
            Priority: 2
            Filter:
              Prefix: ''
            Destination:
              Bucket: arn:aws:s3:::foo-sandbox-ap-southeast-2-public-web
  PublicBucketReplicationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: s3.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: Allow
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                - s3:GetReplicationConfiguration
                - s3:ListBucket
                Resource: arn:aws:s3:::foo-sandbox-us-west-2-public-web
              - Effect: Allow
                Action:
                - s3:GetObjectVersionForReplication
                - s3:GetObjectVersionAcl
                - s3:GetObjectVersionTagging
                Resource: arn:aws:s3:::foo-sandbox-us-west-2-public-web/*
              - Effect: Allow
                Action:
                - s3:GetObjectVersionTagging
                - s3:ReplicateObject
                - s3:ReplicateDelete
                - s3:ReplicateTags
                Resource: arn:aws:s3:::foo-sandbox-*-public-web/*
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions