S3 replication to destination folder

0

Hi Team,

I am trying to achieve the replication from source bucket to destination bucket wherein source and destination buckets are in different account.

condition I have is, can we move the object to destination bucket in a different folder compare to source by using replication rule. if yes how can we do this, if not what all other options can be considered to achieve this task. Please note we are trying to get the data or object from multiple sources into a single destination bucket with replication rule.

Regards, Rajendra

Raj
asked 19 days ago93 views
3 Answers
3

With S3 replication you cannot specify destination bucket's prefix. You can only choose the bucket in any region/account and then replication itself will create the prefix in destination bucket.

If you need objects to be replicated/copy/sync into different prefixes in destination bucket then viable options are :

  1. DataSync, it will allow you to select prefix at destination bucket and can perform copy based on your job schedule (Pls note as it is a copy event, object's time stamp will change, this solution has higher cost compare to replication and its not real time but less management overhead)
  2. Create lambda function to copy objects based on S3 PUT object event, in that function you can define the rules & condition to copy objects to any prefix, this will be more cost affective than Datasync.

I presented a demo of both the solutions to a CX few months ago.

AWS
razguru
answered 19 days ago
profile picture
EXPERT
reviewed 17 days ago
EXPERT
Leo K
reviewed 19 days ago
0

Hi Narravula,

Thanks for your reply. I am able to find the prefix for source while setting the replication rule, whereas I do not find the prefix for destination. Am i missing anything here ?

Regards, Rajendra

Raj
answered 19 days ago
  • You're absolutely right. The destination prefix is not always as explicitly visible as the source prefix when configuring replication rules in the AWS Management Console.

    breakdown of how to locate and set the destination prefix:

    1. Navigate to the Replication Rule:
    • Go to the Properties tab of your source bucket.
    • Under Replication, click on the existing or newly created rule.
    1. Find the Destination Configuration:
    • Look for a section or field that allows you to specify the Destination bucket. This might be labeled as Destination Bucket, Target Bucket, or something similar.
    1. Locate the Prefix Setting:
    • Within the destination configuration, there should be an option to specify the Prefix or Key Prefix. This might be a dropdown, text field, or a separate section.
    1. Enter the Destination Prefix:
    • Input the desired prefix for the destination folder. For example, if you want to move objects to the archived folder in the destination bucket, enter archived/.
0

Hello Rajendra,

Have you explored AWS DataSync for this use-case? Using DataSync, you can copy data between two S3 buckets across accounts, regions, partitions with few clicks and no scripting/coding is required.

With AWS DataSync you can copy data from a prefix in source S3 bucket to another prefix in destination S3 bucket depending on how you configure the DataSync task.

Here is the tutorial that walks you through how to do that: https://docs.aws.amazon.com/datasync/latest/userguide/tutorial_s3-s3-cross-account-transfer.html

AWS
answered 19 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions