S3 for backup of large small files on-premise
A customer wants to use S3 as a backup solution for their on-premise files:
- they have a huge number of small files stored in a NAS (NFS server).
- These files are their customer profiles (json files) and will be updated from time to time
- For performance issue, they set noatime on their NAS, which means we dont have the information about when the file is modified.
Is there any way to help customer to backup the files in an efficient way? S3 sync may help, but may brings two questions : NFS server is not aware of file modification date, does S3 sync work ? Does S3 sync generate large number of S3 request (resulting increase of billing )?
The timestamp atime tells you when the file is last read/accessed. Updating the atime every time a file is read causes a lot of usually-unnecessary IO, so by setting the noatime filesystem mount option you can avoid performance hit. If all you care about is when the file contents change last, then mtime is the timestamp you should be looking at.
Do they have VMware environment on-premises? You may want to take a look at AWS DataSync vs. S3 Sync. There are some advantages over S3 CLI (from our FAQ):
- AWS DataSync fully automates and accelerates moving large active datasets to AWS, up to 10 times faster than command line tools
- It is natively integrated with Amazon S3
- It comes with retry and network resiliency mechanisms, network optimizations, built-in task scheduling, monitoring via the DataSync API and Console, and CloudWatch metrics, events and logs that provide granular visibility into the transfer process
Relevant questions
S3 sync limitations on index size?
asked 3 months agoS3 Intelligent - Tiering - 'Monitoring and Automation' costs
asked 2 years agoMove files from S3 to FSx for Windows
Accepted Answerasked 2 years agoHigh Availability - Network File System (NFS) between EC2
asked 3 years agoDataSync Validate Existing Data in S3 then Copy Changes
Accepted Answerasked 3 years agoSend efs backup to S3
Accepted Answerasked 5 months agoEFS pricing for tiny files
Accepted Answerasked 3 years agoAWS Backup: Backup window
Accepted Answerasked 3 years agoS3-Infrequent Access Retrieval Fee for Range requests.
Accepted Answerasked 4 years agoS3 for backup of large small files on-premise
Accepted Answerasked 2 years ago