DataSync symlink target file from NFS server to EFS

0

I am trying to use DataSync to copy my file from on-premise(NFS 3) to EFS. It says symbolic links are preserved when copying from NFS file server and EFS,(https://docs.aws.amazon.com/datasync/latest/userguide/special-files-copied.html#special-files-copied-symbolic-links). But I want the DataSync to copy the target file of the symlink to the EFS. I've looked into options or datasync cli, but couldn't find a solution.

Any options to suggest?(I have to keep the symlink, but target file destinations are spread out, which makes it hard to get a pattern of them)

3개 답변
3
수락된 답변

If you need to actually follow the symbolic link and copy the data, you can always use rsync with specific options to handle symlinks effectively if AWS DataSync does not meet your requirements.

The -L and -K options in rsync control how symlinks are handled during the synchronization process:

  • -L or --copy-links: This option tells rsync to transform the symlinks into referent files/dirs. In other words, if you have a symlink that points to a file or directory, it will copy the actual file or directory that the symlink points to insteaf of copying the symlink itself.

  • -K or --keep-dirlinks: This option is used with -L to avoid duplicating the referent of a symlinked directory inside a copied tree again at the location of the symlink. This is useful for preventing extra copies of directories already copied elsewhere in the tree.

Combining both options would allow you to follow symlinks for files and handle directory symlinks, specially to avoid unnecessary duplication. Here's how you might use rsync to synchronize an NFS-mounted directory (with symlinks) to a local directory (which you could then upload to an Amazon EFS):

rsync -avzKL /path/to/nfs/mount/ /path/to/efs/mount/

This command:

  • -a: Archive mode; equals -rlptgoD (no -H), preserving almost everything (except hard links) and ensuring that symbolic links, devices, attributes, permissions, ownerships, etc., are preserved.
  • -v: Verbose, which increases the amount of information you're given during the transfer.
  • -z: Compresses file data during the transfer.
  • -K: Ensures that symlinks to directories are treated as the directory they link to.
  • -L: Copies the files that symlinks point to rather than copying them themselves.

You can always create a local copy with all symlinks resolved,and then use DataSync to transfer this data to Amazon EFS.

If the answer is helpful, please click "Accept Answer" and upvote it.

profile picture
전문가
답변함 3달 전
profile picture
전문가
검토됨 한 달 전
profile picture
전문가
검토됨 2달 전
1

You can write a script (e.g., Python or shell script) on your on-premise system that traverses the directory structure and identifies symbolic links. The script can then follow each link, determine the target file path, and copy the target file to the designated location in EFS using a separate tool like aws s3 cp (assuming you're mounting EFS as an S3 bucket). This approach requires some scripting knowledge but gives you full control over the process. Also, Investigate third-party data migration tools like AWS Migration Hub , Gs Richcopy 360, CloudFuze, or CloudEndure Migrate. These tools often offer functionalities beyond basic file copying, including following symbolic links and replicating data structures. However, they might have additional licensing costs compared to using native AWS services.

posike
답변함 2달 전
0

Thanks for answering everyone. As mentioned above, there were no ways to transfer symlink target file using datasync, so I mounted EFS then used rsync. Hope this function will be added in near future.

답변함 2달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠