By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Copy Data from EFS to EFS in Same Account & Region

0

Hi Everyone,

Testing what should be a simple use case in DataSync.

For the moment, just using the console to setup two EFS locations (source and destination) and one task (unscheduled) to copy all data (from task path) from source to destination. Since these two file systems operate in the same region and the same account, I think (hope) I can use a serverless setup that does not require (for example) NFS to EFS with an agent, etc.

I have setup only these resources in DataSync:

  • Location: Source EFS in us-east-1 with correct subnet and SG that has "/" as the mount path
  • Location: Destination EFS location in us-east-1 with correct subnet and SG that has "/" as the mount path
  • Task: in us-east-1 that configures source and destination and mostly uses default settings, though has this notable config:

Data transfer configuration > Specific files and folders > Include patterns > /home/user/subfolder/

Everything looks good and I am able to manually start the task, which moves through the Task status: Launching and then shows Success, but I see that only 1 file is transferred (there are 122 files on the source file system that I am attempting to copy). Looking at the destination EC2 instance, I see only a new hidden directory named: .aws-datasync created in /home/user/subfolder/.

I thought perhaps my include pattern was slightly off and so tested these:

  • /home/user/subfolder/
  • /home/user/subfolder
  • /home/user/subfolder/*
  • /home/user/subfolder*

but the results are always the same: task success but no files (except the hidden dir) transferred ...

Any help is much appreciated.

Ben

Topics
Migration & Modernization
Tags
AWS DataSync
Language
English
bwmills
asked a year ago3423 views
2 Answers
0
Accepted Answer

Transfers between EFS within the same account are supported without requiring the use of an agent. Both the source and destination locations would be specified as the respective EFS filesystems.

https://docs.aws.amazon.com/datasync/latest/userguide/how-datasync-works.html#transfering-files

If EFS does not contain a file system policy you would not need an IAM role to be able to connect to that file system, you would need a role if a file system policy was in place. However, your task would likely fail with a permission issue if this were the case. https://docs.aws.amazon.com/datasync/latest/userguide/create-efs-location.html#create-efs-location-iam

Can you verify from a client where EFS is attached that the /home/user/subfolder/ directories that are used in your include filter are all located under the EFS mount path / that us used in the location?

AWS
Darryl_Dio
answered a year ago
  • bwmills
    a year ago

    Hi Darryl,

    Thanks for your response and for confirming that I don't need an agent and that I should be configuring for EFS (source) to EFS (destination).

    To confirm regarding the presence of a policy on the EFS file systems, I don't have a policy (atm) on either the source or the destination EFS file system.

    I agree that the mount path (path-related configs) is the place to focus. The reason I setup both locations to use a path of / is because when I first attempted to configure the source location with /home/user/subfolder/, the task fails with:

    Task failed to access location loc-... Could not mount subdirectory /home/user/subfolder/ on host .... Please verify that the subdirectory exists and is properly exported in /etc/exports

  • bwmills
    a year ago

    So then I used the config I described in the original post, where both locations use a path of / and the task then uses /home/user/subfolder/ as the include pattern.

    Per your question, I can confirm that /home/user/subfolder/ is mounted as the file system. When I connect to the EC2 instance that runs the application (vsftpd) that needs to access files stored in EFS, df -h shows

    127.0.0.1:/ 8.0E 1.4G 8.0E 1% /home/user/subfolder

  • bwmills
    a year ago

    On that same EC2 instance, I can also cat /var/log/amazon/efs/mount.log

    2022-11-30 21:39:54 UTC - INFO - version=1.34.1 options={'rw': None, 'tls': None, '_netdev': None} 2022-11-30 21:39:54 UTC - INFO - binding 20820 2022-11-30 21:39:55 UTC - INFO - Starting TLS tunnel: "/usr/bin/stunnel5 /var/run/efs/stunnel-config.fs-0c972a4a62d332b4b.home.user.subfolder.20820" 2022-11-30 21:39:55 UTC - INFO - Started TLS tunnel, pid: 2774 2022-11-30 21:39:55 UTC - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ /home/user/subfolder -o rw,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20820" with 15 sec time limit. 2022-11-30 21:39:55 UTC - INFO - Successfully mounted fs-0c972a4a62d332b4b.efs.us-east-1.amazonaws.com at /home/user/subfolder

  • bwmills
    a year ago

    df -T shows: Filesystem Type 1K-blocks Used Available Use% Mounted on 127.0.0.1:/ nfs4 9007199254739968 1409024 9007199253330944 1% /home/user/subfolder

  • Darryl_Dio
    a year ago

    It looks like you have attached EFS to /home/user/subfolder which is the mount relative to your client. DataSync takes the mount path / as a directory directly from EFS. Your include filter would be created to match what is in EFS as visible via your attached mount point. ls /home/user/subfolder   Depending on the contents of the EFS filesystem have you attempted to run this task without an include filter?

0

Can you check the IAM role associated with Datasync and see that it has the right permissions?

AWS
Mike-AWS
answered a year ago
  • bwmills
    a year ago

    Thanks Mike for your response and idea.

    I don't have an IAM role associated with DataSync. Please note that I don't have an agent either (i.e. I think that since it's EFS-to-EFS in the same region/account, I can go with serverless and not have to setup an EC2 instance myself that runs the agent - let me know if I am mistaken on this point).

    Back to IAM - do I need to create a role for DataSync?

    I am not using in-transit encryption.

    Do you know (with EFS-to-EFS in same region/account) whether I should be using NFS (source) to EFS (destination)?

    Thanks, Ben

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content