Datasync from AWS EFS to NFS Server Hosted in Another EC2 Instance

0

I have a EFS instance in Oregon with all the data that I need to sync. The location were I want to store that data could be anywhere, including AWS. For now, we want to make a POC using a EC2 instance in Ireland. We aready setup NFS server where the Datasync Agent should write the files from the EFS, but the problem is that we are not able to create the Datasync Agent in another EC2 instance using the .OVA image provided by Amazon. We tried to import it using this guide: https://aws.amazon.com/ec2/vm-import/.

Is possible to do what I'm trying? To use Datasync to synchronize data from EFS to another NFS server hosted in AWS that emulates the on-premise datacenter, which should be the same as from AWS EFS to On-Premises. If so, how can I install the Datasync Agent image provided by Amazon? If not, which solution should be the one that works for the scenario I'm trying to implement?

PD: The intend of doing this is to have different instances with the same data and if any of them fails, have an instance backup. This is something similar as a fail-over, but we want to keep completely independent data architectures with the same data.

asked 4 years ago97 views
5 Answers
0

amendozaf:
Thanks for your feedback. We don't currently provide the agent for running in EC2.
I've noted your suggestion and we'll consider it as we plan future functionality and features.
Regards,
Paul
AWS Storage Gateway

Paul_R
answered 4 years ago
0

YES please FIX/Add this

FScalzo
answered 4 years ago
0

Dear amendozaf,

You can find instructions for deploying the DataSync agent on EC2 and setting up the desired file transfer here:
https://docs.aws.amazon.com/datasync/latest/userguide/ec2-agent.html

Best,
Olga Kogan
AWS DataSync

answered 4 years ago
0

------ EDIT START ------

THIS IS NOW POSSIBLE. PLEASE CHECK THE COMMENT BELOW POSTED BY OLGA.

I'm leaving the comment just to keep the tracking of the post.

-------- EDIT END -------

Indeed this is not possible yet.This is the Amazon Support Center response:

Hi Andrey,

This is Sayan here. I've read through your case-email. I understand that you want to sync data from your EFS instance in your Oregon region to your on-premise data server in Ireland region. Do correct me if I've misunderstood anything.

Unfortunately, if you want to sync data from EFS to another cross-region EFS instance or another EC2 instance, or if you want realtime sync between EFS to another server, then there is no direct AWS supported feature available to do so currently. Also, the feature of DataSync cannot be used over here. AWS DataSync does not support copying EFS data between different AWS Regions.[1] Now, do note that I have been in constant communication with the EFS team since last few hours regarding whether there can be a service made in the future to realtime sync between EFS to another server. They have said that there is no current feature available for your use-case right now. Having said that, I have logged this as a Feature Request to them. They have not provided any time or guarantee that this is something that will be implemented in the immediate future, but do be assured if this is feasible, then this would be looked upon on high priority.

Now, although there might not be a direct feature for your use-case, but I can suggest 2 workarounds for that :-

  1. You can transfer the data from the EFS to an S3 bucket of the same region. You can use S3 VPC endpoint [2]. These endpoints are virtual devices. They are horizontally scaled, redundant, and highly available VPC components that allow communication between instances in your VPC and services, like S3, without imposing availability risks or bandwidth constraints on your network traffic and should improve transfer speeds from S3 to your VPC and instances. Then, use S3 Cross-region replication featute [3] with source bucket as the Oregon's bucket and the destination bucket as an S3 bucket of the AWS Ireland region. Then download the data from the destination bucket to your own server (AWS hosted ec2 or otherwise). For copying data to and from S3 bucket to the ECS instanes, you can use AWS CLI and the s3 cp (--recursive) and sync commands (e.g., aws s3 sync /path/to/efs/test.txt s3://mybucket/test2.txt). [4],[5],[6],[7]

  2. If you just want the data to be migrated and synchronized (not exactly real-time) between your EFS in Oregon and on-premise server (which is not an EC2 instance and has an NFS agent to it) whether it is in Ireland or wherever else, then you can use AWS DataSync to copy the data from ECS to the On-premise server. [4]
    To get started with AWS DataSync you first deploy a software agent on-premises that is available for download from the AWS Management Console. Once deployed you can use the console or AWS Command Line Interface (CLI) to connect the agent to your on-premises file systems using the Network File System (NFS) protocol. Select your Amazon EFS file system, and start copying data. This is the solution, I would suggest if you are willing to set the destination on-premise NFS file system server to be non-AWS or not an EC2 instance.[8],[9]

Regarding the error you were getting "ClientError: Unsupported kernel version 4.14.77-69.57.amzn1.x86_64" :-
If the instance you're using as the on-premise server (NFS file system) is an EC2 instance, then do note that, currently, DataSync Agent does not support AWS EC2 instance. It supports the following hypervisor versions and hosts:
VMware ESXi Hypervisor (version 6.0 or 6.5 or 6.7) – A free version of VMware is available on the VMware website. You also need a VMware vSphere client to connect to the host.[10] Now, note that this error is generally caused when the requirements are not met with while setting up the agent with on-premise server.

I hope the above discussion was helpful. Please note that for queries regarding EFS setup and VPC, I would suggest you open another ticket/case to that queue/team as they would be the best people to guide you on that. Feel free to reach out to me if you have any additional specific questions on DataSync or require additional clarification on whatever we've discussed.

Have a great day!

REFERENCES

[1] EFS : DataSync - https://aws.amazon.com/efs/faq/#AWS_DataSync
[2] Endpoints for Amazon S3 - https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-s3.html
[3] Cross-Region Replication - https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
[4] Installing the AWS CLI - https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
[5] Configuring the AWS CLI - https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
[6] aws s3 cp - https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html
[7] aws s3 sync - https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
[8] How AWS DataSync Works - https://docs.aws.amazon.com/datasync/latest/userguide/how-datasync-works.html
[9] DataSync FAQs - https://aws.amazon.com/datasync/faqs/
[10] Requirements - https://docs.aws.amazon.com/datasync/latest/userguide/requirements.html

Best regards,

Sayan R.
Amazon Web Services

Edited by: amendozaf on Mar 26, 2019 12:38 PM

answered 4 years ago
0

Thank you very much Olga. I followed your guideline and I was able to implement what I wanted.

Regards,
Andrey Mendoza.

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions