File Storage Gateway temporary storage / option to delay uploads

0

Hi,

So a random one, is there a means to either have a temporary storage area on the the file gateway appliance on prem, or an option to delay file uploads to S3 by a set amount of time?

The rational is that on prem we want to move data to the file gateway via NFS BUT also want to be prudent (as its archive data) and hash the file data after our internal transfer to ensure no corruption occurred as data transferred across our network (overkill - possibly, but the data is really important, so safe rather than sorry - and the average file size is large enough 50 - 100GB that its is a real possibility with some being hundreds GBs to TB range).

Ideally we don't want to have data uploaded whilst hashing just in case there is an issue with it and our systems have to delete and re-copy the data.

If there is a way to delay data being uploaded, or a means to have a temporary storage area no linked to a bucket from which good data can then be copied after the fact that would be awesome. It would also overcome the pitfalls of using rsync and an S3 bucket with version locking (which seems a bit of an oversight).

Thanks in advance,

Owen.

ommps
asked 9 months ago63 views
1 Answer
0
Accepted Answer

Currently there is no provision to delay the upload. Since file gateway can support files larger than the size of the Cache disk, as soon as 5 MiB of data is available, it will start uploading that data.

Your use case is interesting, we will consider that for a future consideration.

answered 9 months ago
  • Thank you for the info. It would be amazing if it could be added as like a system-wide user configurable option in the future for those who need just a bit of time between it hitting the cache disks and starting upload. And as mentioned it overcomes the 'issue' of rsync's temp files and S3 version locking.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions