Re: Move large amount of data from different S3 buckets to different regions with Glacier or an IFA



I have a customer that like to move nearly 1PB of data (OnPrem) to AWS S3 and over half of this is cold archival data. We like to try Datasync but unsure if our new this project will get funding yet.

Also so one recommended using Snowball Edge instead to make it faster as we need to move this all at once but I'm not familiar with this and trying to solution some ideas here. Are we better off using the migration services program for something like this? I believe that the SFTP limit is 5TB from OnPrem to S3 Any suggestions? References

2 Answers

Moving 1PB of data from OnPrem to AWS S3 can be a complex process, but there are several tools and services available that can help you to achieve this.

AWS DataSync is a fast and simple way to move large amounts of data from on-premises to Amazon S3 or Amazon EFS (Elastic File System). AWS DataSync is designed to optimize network utilization and minimize the time it takes to transfer data. It also provides features like data validation, task scheduling, and task monitoring to ensure that data is transferred correctly.

AWS Snowball Edge is a physical device that can be used to move large amounts of data into and out of Amazon S3. It has built-in compute and storage capabilities, and can be used to move large volumes of data faster than over the network. Snowball Edge can also be used to support use cases that require data processing and analysis at the edge, before being transferred to Amazon S3.

AWS also provides several other migration services, including AWS Database Migration Service (DMS) and AWS Server Migration Service (SMS), that can help you to migrate databases and servers to AWS.

Regarding the SFTP limit, it's true that there is a 5TB limit for a single SFTP transfer to Amazon S3. However, you can use multipart uploads to transfer larger files. Multipart uploads can break a large file into smaller parts and upload them in parallel, which can help to optimize network utilization and transfer speed.

Overall, there are several tools and services available that can help you to move 1PB of data to AWS S3, including AWS DataSync, AWS Snowball Edge, and AWS migration services. It's important to evaluate your specific use case and requirements to determine which approach is the best fit for your needs. AWS documentation and best practices can provide further guidance and references on this topic.

Here are some links to AWS documentation that can help you learn more about the tools and services I mentioned:

AWS DataSync:

AWS Snowball Edge:

AWS Database Migration Service (DMS):

AWS Server Migration Service (SMS):

Amazon S3 Multipart Upload:

Additionally, here are some general references on data migration and best practices:

AWS Data Migration Whitepaper:

AWS Data Transfer Costs:

AWS Data Transfer Best Practices:

I hope these resources are helpful! If you have any more specific questions or concerns, please let me know.

Thank you.

answered a year ago

Using the "Storage Class Options" Menu Item

  1. Right-click on the S3 bucket you want to archive to S3 Glacier Deep Archive or an other storage class, then select Storage Class Options item in the context menu.

  2. In the Object Properties box, switch to the Storage Class tab.

  3. In the Use storage class drop-down menu, select the storage class you want your objects to be moved.

  4. When you are finished, click Apply.

Using the “Move to Glacier” Option

  1. Right-click on the S3 bucket that you want to archive to S3 Glacier storage class.

  2. Select Move to Glacier.

  3. In the Bucket Lifecycle dialog box, click Add.. to create a lifecycle rule for your bucket.

  4. In the Bucket Lifecycle Rule dialog box, specify the path and data transition rules.

  5. Select the Enable this rule check box to apply this lifecycle rule to your bucket.

  6. When you are finished, click OK. Note that you can also manage Object Expiration using this dialog box. Click this link to learn more about Object Expiration. Note that the Object Expiration policy may delete files permanently, with no possibility of restoring them.

Note that your backup data may appear to be located in S3 Standard storage class, but in fact it is located in the Glacier storage class. Make sure your backup data is in the Glacier storage class and check the status in the Storage Class column (it should read GLACIER). According to Amazon's Lifecycle policy, it may take up to 24 hours to apply storage class changes.

Using Upload Rules

  1. In the Tools menu, select Upload Rules.

  2. In the Upload Rules dialog box, select the storage account you want to configure, then click Add.

  3. In the Select actions to perform when upload file to cloud storage drop-down menu, select Set HTTP headers.

  4. Specify settings as shown on the screenshot below:Add HTTP header step

  5. Specify file types or buckets you want this upload rule to be applied to, then click OK

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions