S3 Storage Class Migration

0

When I move my data which is in delta format from my S3 standard bucket to glacier through a lifecycle policy the delta logs are also getting inside the glacier which is causing the delta table to fail. Is there any way we can omit the delta logs files and move the rest data automatically through the lifecycle policy. N.B - we cannot use the prefix because the delta log and the data is present in the same folder.

asked a year ago236 views
1 Answer
0

Hi,

If you're unable to move the delta logs to a different prefix (folder), you could consider tagging your objects and then creating a lifecycle rule which limits the scope to the tags you define. That way you could exclude the delta logs by not tagging them, or tagging the differently.

Keep in mind that there are costs associated with applying tags to an object.

Fore more information on object tagging and lifecycle policies, take a look here:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html

https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html#intro-lifecycle-rules-filter

AWS
Tom-B
answered a year ago
  • how can we do that in AWS console...also how can we add tags to omit a specific sub folder under a folder which is going to get archived to AWS glacier

  • To tag objects in the console, you can select multiple objects then hit Actions > Edit Tags. You can't tag sub-folders, and you also can't exclude with lifecycle rules, you can however specify multiple rules which apply to different prefixes.

    This question and answer might help with the questions you're asking here: https://repost.aws/questions/QU7KUrJjJlT6apX8hejMDgfQ/s3-lifecycle-exclude-prefix

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions