Which aws service i can use to continuesly sync data from onpremises to aws

0

I am confused about which service I can use to continuously sync data from on-premises to AWS. I have heard that Datasync will only sync data on a scheduled basis. I want to sync data continuously in the fastest possible way

Sarath
asked 8 months ago731 views
2 Answers
0
Accepted Answer

Syncing data continuously from on-premises environments to AWS can be achieved through various AWS services. A few options include:

AWS DataSync: AWS DataSync can sync data as frequently as every hour. It is fast, secure, and can be used to transfer large amounts of data efficiently.

AWS Storage Gateway: Another service to consider is AWS Storage Gateway, particularly the File Gateway configuration. It provides a seamless way to connect on-premises environments with S3 storage in AWS, offering a virtual on-premises file server, which automatically syncs data to S3.

AWS Transfer Family: The AWS Transfer Family provides support for SFTP, FTPS, and FTP access to S3, which can be used to continuously upload data to AWS from on-premises systems. It's more suitable for file transfers than continuous data synchronization but can be a part of a solution depending on your specific requirements.

AWS DMS (Database Migration Service): If you're looking to synchronize databases, then AWS DMS would be the go-to service. It supports continuous data replication and can be used to migrate databases to AWS with minimal downtime.

Considerations and Best Practices:

  • Bandwidth: Ensure that your on-premises connection has sufficient bandwidth to support the data synchronization requirements.
  • Latency: Be aware of the network latency which might affect the data transfer rate.
  • Data Type: Tailor your solution based on the type of data (file data, database, etc.) you wish to sync.
  • Change Detection: Prefer solutions that support efficient change detection to only sync modified data, reducing the amount of data that needs to be transferred.
  • Security: Ensure data is encrypted during transfer to maintain security.
  • Cost: Be mindful of the costs associated with data transfer and storage in AWS.

There are other services as well, but without more details about what you are trying to do, its hard to really zero in on the right track. That said, I hope this helps.

profile picture
answered 8 months ago
  • The minimum configurable AWS DataSync task schedule is 1 hour [1]. DataSync transfers are task based, tasks can be run manually or on a schedule. DataSync tasks scan files in the source and destination to determine file changes, this prepare phase can exceed minimum task schedules in scenarios where there are 10's of millions files included as part of the task.

    [1] https://docs.aws.amazon.com/datasync/latest/userguide/task-scheduling.html

  • Which is more suitable for continuous data synchronization AWS datasync or Storage gateway?

0

Specifying a schedule in DataSync is optional.

profile pictureAWS
EXPERT
kentrad
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions