Using Location vs Task Filters for Syncing Many Subfolders

Question

Greetings,

In this scenario I have a single on-premises file server per environment. Each file server has a root folder and under the root folder there are 600+ client specific folders.
The idea is to use DataSync to replicate the on-premises files to an identical structure in a target S3 bucket.

My thought is to use the following setup:
* One agent per file server
* One Location for the Source:  on-premises Root Folder
* One Location for the Target: Root S3 Bucket
* One task per client subfolder using filters. There can be large number of files per client, varying of course.

The above scenario would yield in each environment a single Agent two Locations and 600+ tasks with filters for the client subfolder.

To me this seems more rational than creating 600+ locations and 600+ tasks. Does anyone see any potential risks?

*More Details*
* Cheaper to pre-load the data in bulk.
Day to Day replication fee.
DataSync transfer fee: This is a flat fee per GB copied, currently $0.015 per GB in "enhanced mode" for transfers to or from AWS. For 14 TB (14,336 GB), this would be: 14,336 GB × $0.015/GB = $215.04.
Please note: NO Additional charges apply for multiple task requests.

Dhivyaa · Accepted Answer

Hello,

Thank you for reaching out to us via AWS re:Post. Dhivyaa this side from DataSync Support team, and I’m here to assist further on the question you posted here.

From your post, I understand that you have a single on-premises file server per environment, where each server contains a root directory with 600+ client-specific folders. Your goal is to replicate this structure into a target Amazon S3 bucket using AWS DataSync. Please correct me if I’ve misunderstood your requirement.

➨ Few key DataSync Quotas to consider are:
	• Tasks per region per account: 100
	• Max items per task execution (files, directories, or objects): 50 million
	• Max queued executions per task: 50
	• Max characters allowed in a task filter (include/exclude combined): 102,400
	• Agent requirement for >20 million items: Minimum 64 GB RAM for optimal performance

➨ Please note: when transferring S3 objects, DataSync counts each prefix (i.e., folder-like structure) as a directory, which adds to the item count.
[+] DataSync Service Quotas & Limits - https://docs.aws.amazon.com/datasync/latest/userguide/datasync-limits.html#task-hard-limits

- One agent may become a bottleneck if:
	• The file server has very high I/O.
	• Many tasks are running simultaneously.
	• AWS DataSync allows you to throttle bandwidth, but you might need to stagger tasks.

➨ Further, you mentioned that DataSync is charged at a flat rate of $0.015 per GB in Enhanced mode. Just to clarify:
- Enhanced mode is currently supported for:
	• Transfers between Amazon S3 locations
	• Transfers between Azure Blob and Amazon S3 without an agent
	• Transfers between other cloud storage providers and Amazon S3 without an agent
[+] For more details on this, kindly refer: https://docs.aws.amazon.com/datasync/latest/userguide/choosing-task-mode.html

- Since you’re using an on-premises file server with a DataSync agent, Enhanced mode will not apply to your scenario. Instead, your transfers will run in Basic mode, and corresponding pricing will reflect that.
[+] Datasync pricing: https://aws.amazon.com/datasync/pricing/

➨ To stay within these quotas and successfully replicate the structure:

- Use Include Filters: Segment your client-specific folders into logical batches and define include filters for each task to focus on specific folder ranges (e.g., /client001/**, /client100/**, etc.).
- Split Tasks by Folder Groups: If the total item count exceeds the 50 million limit, create multiple tasks—each targeting a different subset of client folders using filtered includes.
- Schedule Task in Batches: You can programmatically schedule multiple task executions in batches or intervals based on priority, size, or business need.
- Use CloudWatch for Monitoring: Track execution metrics, item count, transfer performance, and agent resource usage using Amazon CloudWatch.

Kindly refer to the following resources for more details: 
[+] Scale-Out DataSync Architectures - https://aws.amazon.com/blogs/storage/how-to-accelerate-your-data-transfers-with-aws-datasync-scale-out-architectures/
[+] Using DataSync for Hundreds of Millions of Objects - https://aws.amazon.com/blogs/storage/implementing-aws-datasync-with-hundreds-of-millions-of-objects/
[+] Monitoring data transfers - https://docs.aws.amazon.com/datasync/latest/userguide/monitor-datasync.html

re:Post Agent · Answer

Your approach of using one agent, two locations (source and destination), and multiple tasks with filters is a valid strategy for your scenario with 600+ client folders. This is more efficient than creating 600+ separate locations and tasks.

When transferring large datasets like yours, partitioning the data across multiple DataSync tasks is actually recommended. This helps reduce the time it takes DataSync to prepare and transfer your data. Using filters to target specific client folders is an effective way to partition your data.

For your filter setup, remember that all filters are relative to the source location path. For example, if your source location is the root folder, you would use filters like `/client1/` to transfer only that specific client folder and its contents.

One thing to consider: while your approach is sound, you might want to evaluate if 600+ tasks is the optimal number. You could potentially group some client folders together based on size or other criteria to reduce the total number of tasks while still maintaining efficient transfers.

For large-scale migrations, DataSync allows you to specify different folder paths for each task using include filters, and then run those tasks in parallel. This can significantly accelerate your migration process.

Your cost analysis is correct - DataSync charges per GB transferred, not per task, so creating multiple tasks won't increase your transfer costs.  
**Sources**  
[Transferring specific files, objects, and folders by using filters - AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/filtering.html)  
[Creating a task for transferring your data - AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/create-task-how-to.html)  
[Accelerating your migration with data partitioning - AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/datasync-large-migration-data-partitioning.html)  
[Automate data transfers and migrations with AWS DataSync and Terraform | AWS Storage Blog](https://aws.amazon.com/blogs/storage/automate-data-transfers-and-migrations-with-aws-datasync-and-terraform/)

Using Location vs Task Filters for Syncing Many Subfolders

Add your answer

Relevant content