Unable to move folders within an S3 bucket

Question

Hi,

I'm having trouble moving existing folders to new folders in my S3 bucket, with the following error message: "CopyObjectOperationObject: storage class has unknown or unsupported value : DEEP_ARCHIVE"

I then ran the 'Diagnose with AmazonQ', and followed the instructions on how to add the following permissions to my IAM:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::nsphotobackup/*"
            ]
        }
    ]
}

But that doesn't work either.

Any help would be most appreciated, thanks :)

re:Post Agent · Answer

The issue you're experiencing seems to be related to objects stored in the S3 Glacier Deep Archive storage class. This storage class has some specific limitations when it comes to moving or copying objects.

To address your problem:

1. S3 Glacier Deep Archive: Objects in this storage class cannot be directly moved or copied. They need to be restored first before any operations can be performed on them.

2. IAM Permissions: While the IAM policy you've added includes necessary permissions for standard S3 operations, it may not be sufficient for working with Glacier Deep Archive objects. You might need additional permissions related to Glacier operations.

3. Moving folders: In S3, folders are actually prefixes for objects, not true folders. Moving a "folder" involves copying all objects with that prefix to a new location and then deleting the originals.

4. Size limitations: If any of your objects are larger than 5 GB, you cannot move them using the S3 console. You'll need to use the AWS CLI or SDKs for such operations.

5. Waiting period: When moving folders, especially with many objects, the operation may take some time to complete. It's advisable to wait for the move operation to finish before making additional changes.

To resolve this issue:

1. First, restore the objects from Glacier Deep Archive if they're in that storage class.
2. Use the AWS CLI or SDK for moving large objects or folders with many items.
3. Ensure your IAM policy includes all necessary permissions, including those for Glacier operations if needed.
4. Consider using S3 Batch Operations for moving large numbers of objects efficiently.

If the problem persists, you may need to contact AWS Support for more specific assistance related to your account and configuration.  
**Sources**  
[Copying, moving, and renaming objects - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/copy-object.html)  
[Moving objects to Glacier Deep Archive in S3 | AWS re:Post](https://repost.aws/questions/QUBRt4gAMmQO2-LnBXvn5oSg/moving-objects-to-glacier-deep-archive-in-s3)

Shahed Kazi (@shajam) · Answer

The move options essentially the copies the object which cannot be done when the objects are in Deep Archive.
If you have large number of objects, you can do the following.

* Use lifecycle policy to identify (using prefix or tags) and restore the objects from Deep Archive to Standard.
* Run a batch operation to copy the objects to the new path.
* Write another lifecycle policy to move the copied objects to Deep Archive.
* If versioning enabled, write another lifecycle policy to delete previous versions.

Note that, 
* There will be costs associated with moving the objects between the storage tiers.
* If you have versioning enabled, you will need to delete the versions from the previous path.

Unable to move folders within an S3 bucket

Add your answer

Relevant content