Copy Data from EFS to EFS in Same Account & Region

Question

Hi Everyone,

Testing what should be a simple use case in DataSync.

For the moment, just using the console to setup two EFS locations (source and destination) and one task (unscheduled) to copy all data (from task path) from source to destination. Since these two file systems operate in the same region and the same account, I think (hope) I can use a serverless setup that does not require (for example) NFS to EFS with an agent, etc.

I have setup only these resources in DataSync:

- Location: Source EFS in us-east-1 with correct subnet and SG that has "/" as the mount path
- Location: Destination EFS location in us-east-1 with correct subnet and SG that has "/" as the mount path
- Task: in us-east-1 that configures source and destination and mostly uses default settings, though has this notable config:

Data transfer configuration > Specific files and folders > Include patterns > /home/user/subfolder/

Everything looks good and I am able to manually start the task, which moves through the Task status: *Launching* and then shows *Success*, but I see that only 1 file is transferred (there are 122 files on the source file system that I am attempting to copy). Looking at the destination EC2 instance, I see only a new hidden directory named: `.aws-datasync` created in `/home/user/subfolder/`.

I thought perhaps my include pattern was slightly off and so tested these:

- /home/user/subfolder/
- /home/user/subfolder
- /home/user/subfolder/*
- /home/user/subfolder*

but the results are always the same: task success but no files (except the hidden dir) transferred ...

Any help is much appreciated.

Ben

Darryl_Dio · Accepted Answer

Transfers between EFS within the same account are supported without requiring the use of an agent.  Both the source and destination locations would be specified as the respective EFS filesystems.

https://docs.aws.amazon.com/datasync/latest/userguide/how-datasync-works.html#transfering-files

If EFS does not contain a file system policy you would not need an IAM role to be able to connect to that file system, you would need a role if a file system policy was in place. However, your task would likely fail with a permission issue if this were the case.
https://docs.aws.amazon.com/datasync/latest/userguide/create-efs-location.html#create-efs-location-iam

Can you verify from a client where EFS is attached that the ```/home/user/subfolder/``` directories that are used in your include filter are all located under the EFS mount path ```/```  that us used in the location?

Mike-AWS · Answer

Can you check the IAM role associated with Datasync and see that it has the right permissions?

Copy Data from EFS to EFS in Same Account & Region

Relevant content