Bulk data Migration query from Onprem DC to Cloud

Question

I’m planning to migrate several terabyte  of data from my on-premises data center to Amazon S3. I’ve heard that AWS Snowball Edge devices can help with this large-scale data transfer. Can someone explain how I can use AWS Snowball Edge for this purpose? Also, what are the best practices to ensure data security during the data transfer process?

Akshat · Accepted Answer

Here’s how you can use it and some best practices for securing your data during the transfer:

**Using AWS Snowball Edge for Data Migration:**

1.Create a Snowball Edge Job:
    * Log in to the AWS Management Console and navigate to the AWS Snow Family service.
    * Create a new import job, specifying the amount of data and the target Amazon S3 bucket.

2. Device Shipment:
AWS prepares and ships the Snowball Edge device to your specified location.

3. Set Up the Device:
* Upon arrival, connect the device to your local network and power source.
* Install AWS OpsHub for a graphical interface or use the Snowball Edge client for command-line operations.

4.  Authenticate and Unlock:
* Use the credentials (manifest file and unlock code) provided in the AWS Console to unlock the device.

5. Data Transfer:
* Transfer your data to the Snowball Edge using NFS, S3-compatible API, or AWS OpsHub.
* Monitor the transfer process through logs and dashboards

6. Finalize and Return:
* After the transfer, shut down the device securely.
* Use the provided shipping label to return the device to AWS.

7.  Data Ingestion into S3:
* AWS receives the device and automatically uploads your data to the specified S3 bucket.
* You’ll receive notifications upon completion.

**Best Practices for Data Security:**

**Encryption at Rest and In Transit:**
	•	Data is automatically encrypted using 256-bit encryption keys managed by AWS Key Management Service (KMS).
	•	The encryption keys are never stored on the device.

**Secure Credentials Handling:**
	•	Keep the manifest file and unlock code secure; they are essential for accessing the device.
	•	Do not store these credentials on the same machine you’re transferring data from.

**Access Control**
	•	Use AWS Identity and Access Management (IAM) to restrict who can create and manage Snowball Edge jobs.
	•	Implement role-based access control in your organization.

**Physical Security:**
	•	The device is tamper-evident and tamper-resistant.
	•	Track the device using AWS SNS notifications and shipping company tracking.

**Data Verification:**
	•	After the data is ingested into S3, verify the data integrity using checksums or AWS S3’s object versioning and MFA Delete features.

**Network Security:**
	•	Connect the Snowball Edge device in a secure network segment.
	•	Ensure that firewalls and security groups allow necessary traffic for data transfer.

Didier_Durand · Answer

Hi,

Please, be aware of a very recent update about Snowball Edge: https://aws.amazon.com/blogs/storage/aws-snow-device-updates/

Some models are discontinued.

The preferred way to do transfers nowadays is "over the wire" with solutions like AWS Datasync: https://aws.amazon.com/datasync/

```
Since its launch in 2015, customers have used AWS Snow devices to move data to the 
AWS Cloud or run compute and processing workloads at the edge. Our innovations since 
have made moving data to AWS and running workloads at the edge, faster, more efficient, 
and more cost effective. During the same time, network bandwidth has increased exponentially, 
and we have introduced services such as AWS DataSync, a service that automates and accelerates 
moving data between on premises and AWS, and customers have taken advantage of the network 
speed increases to move tens of petabytes of data to the cloud using AWS Direct Connect, a service
 that delivers consistent, low-latency performance for moving data to AWS by linking your network.
```

If your datacenter is connected to the AWS cloud at 1 GBps (via the internet of directly) and your transfer params in Datasync are optimized, a 1TB transfer is supposed to take approx 2.5h.

See https://expedient.com/knowledgebase/tools-and-calculators/file-transfer-time-calculator/ for your own estimation

AWS DataSync supports restarts in case your transfer breaks in the middle.

Best,

Didier

Bulk data Migration query from Onprem DC to Cloud

Relevant content