Questions tagged with Amazon S3 Glacier
Content language: English
Sort by most recent
S3 to Instant Retrieval Glacier
Hi I have setup a lifecycle rule to target a specific prefix (collection of files) in s3 to change storage class to Instant Retrieval Glacier but some items are not changing storage class. The file size varies to what is not moving and i have not excluded any specifics. It was also set to zero days to move. Some of the files are small others not. It has been over 48hours (7 days) sine the lifecylce rule had been created.
Multi Download from Amazon S3 (Glacier Deep Archive)
Hello Community, I am having an issue with downloading multiple files from S3. My files (photos) were stored in Intelligent Storage and made their way all the way down to Glacier Deep Archive. I have now restored the files and I can download them individually, but since there are thousands of photographs I do not want to do this one by one. I followed some steps to use the AWS CLI to download the photos: ``` aws s3 cp --recursive s3://<bucket>/<folder> <local_folder> ``` As well as just to copy it to a new S3 bucket ``` aws s3 cp --recursive s3://<bucket>/<oldfolder> s3://<bucket>/<newfolder> ``` The error message I see for both of the above is ``` Object is of storage class GLACIER. Unable to perform copy operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 copy help for additional parameter options to ignore or force these transfers. ``` So although all of the files have been restored, they are still showing as GLACIER when I try to copy them using the CLI, but I can download them instantly one by one using the UI. Please can you advise how I will be able to bulk download these files? Thanks, Billy
archiving directly to glacier
We are developing a glacier archive solution, but now wondering whether it would be better to upload to S3 then use lifecycle rules to move the files to glacier. We would be uploading terrabytes of data every day which include files that could be 50G and files that could be 1M. Now wondering if anyone actually goes straight to glacier or if that is not really a thing any more.
HITs - MTurk Workers Upload Problem with AWS
Hello, I tried to do everything in this tutorial: https://blog.mturk.com/tutorial-how-to-create-hits-that-ask-workers-to-upload-files-using-amazon-cognito-and-amazon-s3-38acb1108633 But in the end, when the MTurk workers try to upload their work, they get this message: "Failed to upload. Please contact the Requester." I don't know what is the problem. Please help.
Does S3 Inventory include Overhead, Staging or Restore Object data?
Hi, From my analysis, I couldn't find in S3 Inventory any of the data related to the `Overhead, Staging or Restore Object` ITEM OPERATIONs. I've been only able to find data that has the following Item Operations: `DeepArchiveStorage, GlacierStorage, OneZoneIAStorage, ReducedRedundancyStorage, StandardStorage`. S3 Inventory only provides data about the Storage Class. But I compared the data from S3 Inventory with other reports where we have the data detailed by Item Operation, and the missing data from S3 Inventory seems to be `Overhead, Staging or Restore Object` ITEM OPERATIONs. Is this correct to assume this is the expected behaviour? Thank you!
S3 Object Lock and incomplete uploads.
Say you want to upload a bunch of really large files. However somewhere in between the upload something goes wrong. The files don't complete. The whole thing is a mess so you just want to start over. If you set up the bucket with Object Lock in Compliance mode. You're pretty much just going to have to eat the costs for the durations of the retention period? Am I understanding this correctly? Do you guys have any advice for dealing with these situations? This is for backups I am currently using Amazon S3 (not Glacier). Perhaps I should be using Amazon S3 Glacier with Vault lock instead. But I'm not as familiar with it as I've never used it before. Is there a better way to handle this situation? Ultimately I want the files to be immutable after I upload them. But I wouldn't mind if it waited until the upload finished. (SEPARATE QUESTION) Another problem I run into is my S3 Browser uploads everything as "Standard". When I change everything to Glacier Deep Storage it redownloads the data again. While also creating a separate version still in Standard. Am I going to be charged for both? And if so is there any way to avoid this besides using the funky browser upload? (That is how I got the messed up original upload) So in summation: 1) How does Object Lock handle incomplete uploads. Is there anyway to delete them after you uploaded them. 2) Is there a better way to apply a different storage tier to an Object Locked S3 Bucket. Besides uploading it twice or using the browser. Thank you!
Datasync to copy data from one S3 bucket to another S3 in the same account
Hi All, We are looking to copy approx 500 TB of Data from one S3 bucket to another S3 bucket in the same region. Do you think that DataSync is the fastest and best available option for the transfer? And how much time will it it approximately to copy 500 TB of data? If we have to copy 7 folders, do we have to create the 7 tasks and can this impact the maximum throughput which 10 GBPS. Thanks Rio
Does Inbound Data Tranfer means "Upload to AWS" and Outbound means "Download"?
I'm a student and I'm new to AWS. I learned that Inbound Data Transfer is all free while Outbound is charged. My question is what does Inbound/Outbound Data transfer actually mean? Does Inbound = Upload data from my PC to AWS Storage/Database and Outbound = Download from AWS back to my PC? I found a similar question on StackOverFlow but I still want to confirm it here https://stackoverflow.com/questions/21320063/what-is-inbound-outbound-data-transfer
DMS : Replication Task is in 100% running state, but stopped replicating data in between. Source - RDS, Destination - S3.
DMS replication task, which has source as RDS - MySQL and destination as S3, it is ongoing cdc replication task and is showing status as runnning 100%, for 4 hours it was not replicating any data, when we checked RDS, it was having new data of those 4 hours, so why did DMS didn't and also didn't put error status. Later we just stopped and resumed, 3-4 times and then after 3rd attempt data started coming from 4 hours before checkpoint. This issue has happended for our omnifin database (comparitively big Db, issue happened for the first time) and for knab database (very small Db, on every 27th day of the month this issue happens) So why does DMS without showing any error, stops performing its replication task, and in status it is green with runnning 100% display. Please look into this issue. Thanks.