By using AWS re:Post, you agree to the Terms of Use
/AWS Storage Gateway/

Questions tagged with AWS Storage Gateway

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

'AWS suggested' way to deal with object lock, glacier, and File Gateway (ie lifecycle logic / lamdbas)

I'm interested (somewhat linked to other questions about how uploads to the Storage Gateway in File Gateway mode work when there is a need to hash data after copying to the NFS share) in the 'AWS suggested' ways to handle the mix of File Gateway, Object locking, and Deep Glacier transitions because it feel like its easy to set up wrong, and treble storage costs inadvertently. I'm noticing for each upload, two object versions are made. I read ([Here](https://docs.aws.amazon.com/storagegateway/latest/userguide/troubleshooting-file-share-issues.html#s3-object-versioning-file-gateway-issue)) this is because of the way data is written to NFS where the file metadata is transferred after the file contents (which the storage gateway would have already started to upload). So Before I even start looking at Lifecycle rules I already have two object versions for each file (they both seem to be of the same size, so doubling storage costs out the bat)...... not ideal... I then want to use a lifecycle rule to move data to Deep Glacier. But then I have one object version in Deep Glacier, and one version in S3 'Standard' (increasing storage costs).... I want to take advantage of object locking to protect from ransomware 100% (that is after all one advantage of WORM). However, because I now have multiple object versions I need to have lifecycle rules to delete the older versions to save on costs, BUT, if you ONLY keep the latest version, there is a significant risk of ransomware encrypting data waiting to move to Deep Glacier (as lifecycle rules only run once per day). This would create a new 'encrypted' object version, which then potentially gets moved to Deep Glacier as 'the most recent' object, and old versions (ie the un-encrypted version that represents the actually object WE uploaded and is not affected by the ransomware) are then removed leading to a mix of ransomware affected objects in Deep Glacier with no way of knowing which is and isn't... I hope that scenario makes sense? We cannot just blindly keep the most recent object version, as it could have been affected by ransomware, but we cannot, and do not, want to keep a million object versions (of which a min. of two object versions are always made because of the way NFS metadata is written 2nd)... I want to (as a side note) put legal holds onto Deep Glacier objects just because we effectively want indefinite retention, but still have the ability to remove and delete as needed when situations arise. How do AWS envisage this is supposed to work? I would appreciate some insight as to the types of lifecycle policies, and or lambda functions that would be needed to purely from a 'system logic' perspective (ie rough ideas of what needs to be done, I'm not asking for documented examples - unless they already exist). Owen. PS. not having versioning and object locking doesn't help the main aim of protecting the data from ransomware / malicious code as another way to destroy this 'archive data lake' would be to rm -rf from the NFS mount.... which without the object locking / versioning would delete all the data.....
1
answers
0
votes
4
views
asked 2 months ago

Understanding changes with Glacier Deep Archive

I've had a Storage GatewayVTL for a while and it's been great. We use Veeam as the backup application, FWIW. I am now trying to figure out exactly what the changes are now with Glacier Deep Archive. In the past, my understanding was that anything uploaded to the VTL, by default, was stored in S3 (not S3 Glacier, regular $.023GB/month. If we wanted to move something into Glacier, we would either set the job in Veeam to automatically export the tape after the job completed, or manually export the tape later. I have done this many times, and those virtual tapes would show up in the AWS Storage Gateway console as "archived". And our bill would reflect that, as far as I could tell based on storage used etc. Now, in the SG console, there is this new (I think it's new, I don't remember seeing it before) column for "Pool". There is still an "Archive" status column that my previously archive tapes still have a "Date archived" value that is consistent with past status. I am finding that I can create new tapes and select which pool I want to put them in, but the only options are Glacier and Deep Archive, no regular S3. This would seem to indicate that regular S3 is no longer part of the picture with a VTL, it's either Glacier or Deep Archive. However, the "updated" pricing for Storage Gateway still references S3 as part of the model. Also, all of my tapes currently are listed under the "Pool" column as being in the "Glacier Pool". I've not found anywhere the states how/if S3 is used for virtual tapes and was wondering if anyone could shed some light on it. Thanks!
1
answers
0
votes
0
views
asked 3 years ago
  • 1
  • 90 / page