Skip to content

Change to Glacier Deep Archive storage class? Can no longer use this class for files > 5GB

0

Hello,

I have been using Glacier Deep Archive S3 storage class to archive data for several years now. These particular files include raw data that varies from the 2-20GB range, and some additional metadata that's more in the MB range.

I have a bucket where I have created several folders to organize my data. Generally I sync data to these folders and then Edit Storage Class and select Glacier Deep Archive.

The latest time I did this, some data correctly change storage class but for other files I got a new error message "The object exceeds the maximum size of 5GB". Is this a new change to the Glacier Deep Archive class?

Looking at another folder I synced just a month ago, I have many files >5GB that are stored as Glacier Deep Archive. I wasn't able to find any information on file size restrictions directly in the documentation (https://docs.aws.amazon.com/AmazonS3/latest/userguide/glacier-storage-classes.html#GDA).

I'm looking for a way to resolve this issue and backup this folder fully as Glacier Deep Archive, to avoid excessive storage costs. Or alternatively I am looking for documentation on changes to the Glacier Deep Archive storage class to share with management and justify new storage expenses.

Thanks for your help!

3 Answers
2

I apologize, but there seems to be a misunderstanding in your question. There has not been any recent change to the S3 Glacier Deep Archive storage class that would limit file sizes to 5GB. S3 Glacier Deep Archive continues to support objects of various sizes, including those larger than 5GB.

The error message you're encountering is not related to a limitation of the S3 Glacier Deep Archive storage class itself. Instead, it's likely related to the method you're using to change the storage class of your objects.

When changing the storage class of existing objects in S3, there are different methods available, and some of these methods have limitations:

  1. Using the S3 console to change the storage class has a limitation of 5GB for individual objects.
  2. Using the AWS CLI or SDKs, you can change the storage class of objects larger than 5GB.

For objects larger than 5GB, you should use the AWS CLI, SDKs, or REST API to change the storage class. These methods support changing the storage class for objects up to 5TB in size.

To resolve your issue and backup your folder fully as Glacier Deep Archive:

  1. Use the AWS CLI or an SDK to change the storage class of your larger files.
  2. For future uploads, consider setting up a lifecycle rule to automatically transition objects to Glacier Deep Archive after a specified time period.

Rest assured, S3 Glacier Deep Archive remains a viable and cost-effective option for long-term storage of large files. There's no need to justify new storage expenses to management due to size limitations, as the storage class itself hasn't changed in this regard.

If you continue to experience issues, I recommend contacting AWS Support for further assistance, as they can provide more specific guidance based on your account and configuration.
Sources
Understanding archival storage in S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive - Amazon Simple Storage Service
Coming Soon – S3 Glacier Deep Archive for Long-Term Data Retention
Understanding S3 Glacier storage classes for long-term data storage - Amazon Simple Storage Service

answered a year ago
0

Interesting. I have always changed storage class using the S3 console without issue before, even for files >5GB. I will try using the CLI. Thanks.

answered a year ago
0

You can move an object if your object is less than 5 GB. If your object is greater than 5 GB, you must use the AWS CLI or AWS SDKs to move an object.

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/copy-object.html#MovingObjectsExamples

You can change an object's storage class using the Amazon S3 console if the object size is less than 5 GB. If larger, we recommend adding an S3 Lifecycle configuration to change the object's storage class.

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/sc-howtoset.html#changing-storage-class

AWS
EXPERT
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.