Please help me choose the right Amazon service(s)

Question

I have a customer that collects terabytes of read-only "discovery" data from legal cases, like body cam footage and computer/device dumps. I would like to keep all the files the company creates on their internal Windows server and move all the discovery data to cloud storage. I will need to move new discovery data from their Windows server to cloud storage on a regular basis and users will occasionally need read-only access the discovery data from their Windows computers.

I have been learning some AW services, like creating S3 buckets, creating EFS, and FSx so I have a general understanding but I can't figure out which best suits the requirements. EFS looked good but I read it only works with Linux. FSx also looked good but it has a 65TB limit so it doesn't make a good storage server. Perhaps the solution is a combination of services.

What AW service or services would you use to meet these requirements and how do you see them working together?   Thank you in advance!

Debapriyo · Accepted Answer

For your scenario, a combination of AWS services can effectively meet your requirements for storing terabytes of read-only discovery data while ensuring regular access and easy transfer from your internal Windows server. Here’s a recommended approach:

### 1. **Amazon S3 (Simple Storage Service)**

**Usage:** Use S3 for storing all the discovery data. It is designed for scalable object storage, making it ideal for large amounts of read-only data like body cam footage and computer/device dumps.

**Benefits:**
- Virtually unlimited storage.
- Cost-effective for large datasets.
- Data can be accessed via the AWS SDKs or AWS CLI, making it easy to transfer files from your Windows server.
- Built-in redundancy and durability.

### 2. **AWS Transfer Family**

**Usage:** Set up AWS Transfer for SFTP (Secure File Transfer Protocol) to facilitate secure file transfers from your Windows server to S3.

**Benefits:**
- Users can transfer files securely without needing to manage any servers.
- Simplifies the transfer process and allows for regular updates to the S3 bucket.

### 3. **AWS Storage Gateway or Amazon S3 File Gateway**

**Usage:** Consider using the File Gateway option of AWS Storage Gateway. This will allow your Windows server to interface seamlessly with S3.

**Benefits:**
- Provides a local file system interface to S3, so your applications can continue using standard file protocols (like SMB).
- You can regularly sync data to S3 without changing how users interact with files on their Windows machines.

### 4. **AWS Identity and Access Management (IAM)**

**Usage:** Use IAM to manage access permissions for users who need read-only access to the discovery data in S3.

**Benefits:**
- Fine-grained control over who can access the data and how they can interact with it.

### 5. **AWS CloudFront (Optional)**

**Usage:** If you want to enhance access speed or provide public access to certain data, consider using CloudFront in front of your S3 bucket.

**Benefits:**
- Global content delivery with caching for faster access.
- Helps reduce latency for users needing to access data remotely.

### How They Work Together:

1. **Data Transfer:** Users upload new discovery data to their internal Windows server. AWS Transfer for SFTP securely transfers this data to S3 on a scheduled basis.

2. **Storage:** All discovery data resides in S3, providing virtually unlimited storage. The data can be structured using folders for better organization.

3. **Access:** Users access the data directly through their Windows machines via the AWS Storage Gateway, which presents the S3 bucket as a local file system. This way, they can easily read and interact with the data without needing direct S3 access.

4. **Security:** IAM roles ensure that users have the necessary permissions for read-only access, keeping the data secure.

In **Summary** Using **Amazon S3** for storage, **AWS Transfer Family** for secure transfers, and **AWS Storage Gateway** for seamless integration with your Windows environment provides a robust solution. This architecture ensures that you can efficiently manage large volumes of data while maintaining ease of access for users.

Here are some useful AWS documentation links that provide more information on the services mentioned:

1. [Amazon S3 Documentation](https://docs.aws.amazon.com/s3/index.html)

2. [AWS Transfer for SFTP](https://docs.aws.amazon.com/transfer/latest/userguide/what-is.html)

3. [AWS Storage Gateway Documentation](https://docs.aws.amazon.com/storagegateway/latest/userguide/what-is-storage-gateway.html)

4. [Policies and permissions in Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html)

5. [Amazon CloudFront Documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html)

These resources will help you understand how to set up and use these services effectively for your use case.

Riku_Kobayashi · Answer

Hello.

Will the total data be more than 65TB?  
In that case, I think the option would be S3.  
In the case of S3, there are several ways to access it from on-premises, but by using the S3 File Gateway, which is a feature of Storage Gateway, it is possible to send and download files from a Windows server to S3 via SMB.  
https://docs.aws.amazon.com/filegateway/latest/files3/what-is-file-s3.html

Please help me choose the right Amazon service(s)

1. Amazon S3 (Simple Storage Service)

2. AWS Transfer Family

3. AWS Storage Gateway or Amazon S3 File Gateway

4. AWS Identity and Access Management (IAM)

5. AWS CloudFront (Optional)

How They Work Together:

Add your answer

Relevant content