Questions tagged with AWS DataSync
Content language: English
Sort by most recent
Hi guys,
I'm trying to set up data sync between 2 EC2 instances.
I'm setting up the locations, setting up mount to /
Then I create the task, specifying the folder I would like to move. Tasks run, and it shows completed, but no new data appeared on my target instance. On the task, I see only one file was moved, which I can't find.
Does anyone have any idea how to set it up?
I'm following this tutorial: https://docs.aws.amazon.com/datasync/latest/userguide/tutorial_s3-s3-cross-account-transfer.html#s3-s3-cross-account-create-datasync-destincation
The steps have been completed accurately through step 3. When attempting step 4, I get the error
```
An error occurred (InvalidRequestException) when calling the CreateLocationS3 operation: DataSync location access test failed: could not perform s3:HeadBucket on bucket name-of-my-bucket. Access denied. Ensure bucket access role has s3:ListBucket permission.
```
I've ensured the bucket access role has the s3:ListBucket permission, and I don't know what to do next.
Hi,
Is there a way to calculate/estimate how long it will take for AWS Datasync to migrate data from on-prem to AWS? I know internet condition and disk I/O will affect the transfer time, but is there a general estimate we can get assuming all "under normal conditions"
Example 10TB data - max file size 500KB - Bandwidth 1Gbps
Thank you.
Hi,
Can I migrate data on tape archive on-prem to AWS using AWS Datasync? This is a complete migration, NOT extending your storage. The amount of data is 10TB with max 500KB file size, bandwidth 500Mbps. I know there is the option of Snowball but on this case I want to know more about Datasync. Once agent is installed on the on-prem VM, can it discover and read the tape storage with NFS protocol or further configuration are required to present the tape storage to the Datasync Agent?
Thank you,
Amanuel.
Hi,
I have tried to synchronize FSx Lustre with S3 but I have the following error :
Failed to read next entry while processing directory /: Skipped S3 prefix not supported by DataSync. Please contact AWS support in order to identify the prefix.
There is no files in .aws-datasync/ and the DataSync task use 2 locations : S3 and FSxLustre with the root directory / . When I launched the task, I specified the parameter include and select the option (Verify only the data transferred).
I have tried to change compression/ Root Squash in Lustre but nothing seems to work.
Do you experiment this trouble?
Regards,
Vivien Moretto
In need to transfer files(NFS mount paths) from on-premise to AWS EFS.
In the document, I saw an option of having the DataSync agent on EC2 machine, so can we ssh AWS EC2 machine at on-premise side and transfer the NFS files from on-prem to EFS?
what could be the detailed steps involved in this?
Hello, I am having an issue with Elastic Disaster recovery agent.
Everything was working fine and the machine was fully replicated to AWS, However all of a sudden I got the error that AWS Agent is not seen.
And a Lag of 3days with no other logs. Can anyone help with this ?
I am trying to connect SMB or NFS folder but in both I am getting this error.
Task failed due to a transient error. Please re-run the task and contact AWS support if the problem persists
and in CloudWatch :
[INFO] Execution exec-0000000000 finished with status Ping received for wrong hostId.
I installed VM on HyperV.
Connectivity to FS and Network both are working. PASSED
There is no mounting or authentication issue still getting same above error.
Please advise.
Thank You
Hi
Im using AWS Datasync agent for data migration b/w on-premise (without direct connect or AWS VPN) and aws data center, I was using linux machine to deploy KVM in Virtual manager. (Using Public service endpoint) When the agent is activated and connection is established, the agent is online that time alone after minute, its going offline. Even though the network connectivity is passing. If someone can help me here, would be grateful.
PFA 
Looking forward for some solution/idea to debug this issue.
Thanks in advance
Hi Friends,
I have a stock broking customer. He has a database. I need to pull data out of that and put in my solution and build reports.
The design I am thinking is :
1. The customer posts the data to SQS.
2. I deploy a Lambda which process the SQS and stores the data in AWS database.
Why I need SQS is reduce the post call latency. He will post some 10K records one after the other and it should not be blocking for him.
Now which DB is best for my use case ? Is it Aurora or RDS or Dynamo DB ? What are the differences ? Kindly share your insights.
Thanks,
Sooraj Sairam Aanand
We have a requirement to sync the data from the on-prem database to AWS RDS (PotgreSQL) at specific intervals (unlike one-time data migration). Assume there is an Interconnect/VPN already established between AWS and On-prem network. The expected data volume is likely 1000 rows only, so I do not see the necessity to build ETL with AWS Glue. Given that, what are the possible solution options to fetch the data?
Can AWS Batch/a pg_cron job be considered here to execute a set of select and update SQLs?
Alternatively, how do we trigger the AWS Lambda at certain intervals if AWS Lambda is a solution option for this requirement?
Appreciate your input.
How may I create / run tasks in "what if" mode? Essentially I would like to generate a report of what might occur if I rum a task without actually making any changes to either locations. Source and destination locations are SMB server and FSx for windows file server respectively.