- 最新
- 最多得票
- 最多評論
This process is not directly supported by OpenZFS and requires a custom solution. Here's a high-level approach to automate the migration of data older than a specified period, such as 6 months, to Amazon S3 Glacier:
- Identify Older Files First, use tools and scripts to identify files older than 6 months. You can use the find command in Unix-based systems to list these files:
find /path/to/zfs/dataset -type f -mtime +180
This command lists files modified more than 180 days ago.
- Archive and Transfer Before moving files to S3 Glacier, consider archiving them to reduce the number of objects and possibly save on costs. You can use tar or other compression tools for this purpose:
tar -czvf archive-name.tar.gz /path/to/older/files
- Upload to Amazon S3 Glacier You can use the AWS CLI to upload the archived files directly to an S3 bucket configured for Glacier storage:
aws s3 cp archive-name.tar.gz s3://your-bucket-name/path/to/archive/ --storage-class DEEP_ARCHIVE
The DEEP_ARCHIVE storage class offers the lowest cost storage option in S3 but with a retrieval time of 12 hours or more.
- Automate the Process To automate this process, you can create a script that performs these steps and schedule it to run periodically using cron jobs or other scheduling tools.
As the question is tagged with Amazon FSx for OpenZFS what follows assumes that's where the data is located that needs to be migrated (and not, say, a third-party on-prem OpenZFS product) then AWS DataSync is the way to go.
https://aws.amazon.com/datasync/faqs/#Data_movement
Q: Where can I move data to and from?
A: DataSync supports the following storage location types: .... Amazon Simple Storage Service (Amazon S3), .... Amazon FSx for OpenZFS file systems
Even if your data is currently on-prem it may still be worth looking into.
Q: How do I use AWS DataSync to migrate data to AWS?
A: You can use AWS DataSync to migrate data located on premises, at the edge, or in other clouds to Amazon S3
The above mentions "plain" S3, but Glacier also gets a call-out in the same section of the FAQ.
Q: How do I use AWS DataSync to archive cold data?
A: You can use AWS DataSync to move cold data from on-premises storage systems directly to durable and secure long-term storage, such as Amazon S3 Glacier Flexible Retrieval (formerly S3 Glacier) or Amazon S3 Glacier Deep Archive.
相關內容
- 已提問 1 年前
- AWS 官方已更新 1 年前
- AWS 官方已更新 2 年前
I had checked DataSync, while it allows moving data between FSx and S3 (I did not test it), I did not find any option to specify a rule. My requirement is to not just move data between FSx and S3 but archival of files greater than a certain age. Please let me know if my understanding is incorrect. Thanks
I haven't tried it myself either, according to https://docs.aws.amazon.com/datasync/latest/userguide/create-s3-location.html#using-storage-classes
The steps to do create the S3 transfer location & specify the storage class are at https://docs.aws.amazon.com/datasync/latest/userguide/create-s3-location.html#create-s3-location-how-to