- Newest
- Most votes
- Most comments
Moving 20TB is not the problem, in this case not even the average size of the files. The problem posted is that someone or even a script, has to crawl directories to set permissions on the target. I can't see a solution for this to quicken.
Hello!
If files will have individual permissions there is no way than crawl all of them. Ideally, you'd have a structure that allows for folder level permissions that would translate to prefixes in S3, you can then set access permissions per prefix.
I hope this helps.
Hi there.
Regardless of the time it will take to configure doing this over the Internet or via an AWS Direct Connect channel, 20TB is a lot of information and will take a lot of time to move. I would recommend you check AWS Snowball and use a Snowball Edge device to copy this data into AWS. It will get directly imported into your S3 bucket.
Here is how AWS Snowball Edge works.
I hope this helps.
As I read it, the problem is not the transport of bytes, but of permissions. Permissions come in two main flavours; local (in the filesystem/bucket/object) or externally (like AD or AWS directory services). Therefore you will need a service which A. can read existing permissions per file/directory and B. can store comparable permissions along with the file/directory.
p.s.1 20TiB equals 20,480 GiB, which equals 204,800 giB (small 'g'), which easily can be compared along a 10Gbps or 100MBps network. p.s.2 High transfer speeds are possible with block storage, but for file/objects, the amount (number) of these is of much more influence).
BR, Henk
Relevant content
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
Sorry to say, but this might be painful and yes, will cost many hours. What is your use case for the migrated files? If you can, arrange permissions from the highest level possible. Some files won't need to be accessed at all (e.g. archived)?
And even a smart script will have to crawl all directories to read the current permissions and apply them to the migrated directories.
I want to use hybrid cloud solution. Storgate Gateway Servers will have 3TB storage for caching frequently access file. The only concern about solution to reassign permission after migrating (a lot of department, complex directory structure)