To restore S3 objects from source to destination bucket

0

Hi AWS, this question is an extension to question https://repost.aws/questions/QU8dBP1TBCTsaDcSTCK_CfDQ/to-transfer-s3-glacier-objects-to-s3-bucket. We have a bucket in region 'eu-west-1' where the objects Storage Class is Glacier Flexible Retrieval (Formerly Glacier) to a destination bucket in the same region having Storage Class Standard. I tried this aws cli command

aws s3 sync s3://source-bucket/img.jpeg  s3://destination-bucket

The command executed successfully but there was no object transferred to the destination bucket and it is totally empty.

Secondly I was having a thought to change the storage class under the Actions tab to Standard Storage Class first as mentioned in the screenshot attached S3 Storage Class and then copying it to the destination bucket.

Other option I was thinking of is a need to initiate restore and then need to change the Storage class and further copying it to destination bucket using S3 Batch Operations as mentioned in this blog post https://repost.aws/knowledge-center/s3-batch-operation-initiate-restore

Or furthermore do I need to follow any of these documentations link: https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects-retrieval-options.html

Please acknowledge and provide me the solution for the same.

profile picture
asked 11 days ago218 views
3 Answers
3

Hello, yes you can achieve this but it can restore after a pre-defined period of time. we have to create a Lifecycle Rule for this.

Through console it is bit easier to choose options but,

If you want to create through CLI Reference : https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html

Follow this steps:

  • First write0 a JSON file to create a Lifecycle Rule
{
  "Rules": [
    {
      "ID": "TransitionGlacierToStandard",
      "Prefix": "archives/",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
    }
  ]
}

  • Make sure to change the configuration terms according to your terms (ID - unique identifier for your Rule, Prefix - Prefix for objects to apply rule, Status - Enabled or Disabled, Transition - Specific time period).
  • After that save the file.
  • Use this command in CLI **"aws s3api put-bucket-lifecycle-configuration --bucket your-bucket-name --lifecycle-configuration file://lifecycle-rule.json " ** Note : Make changes of Your Bucket name and JSON file name and its path.
profile picture
answered 11 days ago
  • Hi Mahaboob Basha, so after X number of days once the objects will be transitioned to STANDARD from GLACIER then we can copy the objects from source bucket to destination bucket, right?

0

Hello,

Yes, you would need to restore the objects from the Glacier storage class before you can make any changes to them.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html#sc-glacier

  • S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects are not available for real-time access. You must first restore S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects before you can access them.

If you have large number of objects, yes, I recommend using S3 batch operations.

psp
answered 11 days ago
0

Hi

The command aws s3 sync s3://source-bucket/img.jpeg s3://destination-bucket doesn't work for transferring a Glacier object because S3 Glacier objects are archived data and not readily available for retrieval. They are meant for long-term, low-cost storage.

  • Glacier Object Accessibility: S3 Glacier objects are in a special storage class designed for low-cost archiving. This means they aren't immediately accessible for retrieval like standard S3 objects.
  • aws s3 sync Behavior: The sync command works by comparing the source and destination and copying objects that are missing or have changed. However, in this case, the sync command can't "see" the Glacier object because it's not readily available.
  • You need to restore it https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html

https://repost.aws/knowledge-center/s3-batch-operation-initiate-restore

profile picture
GK
answered 11 days ago
profile picture
EXPERT
reviewed 11 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions