Run multiple Datasync jobs in parallel

Question

| Run multiple Datasync jobs in parallel  |
|----|
I have an agent installed in my data center that is located in a private network. I can start one job at a time to move the data from on prem to s3. If I started another job it's queued (waiting for the first job to be completed). I believe this is not the way it should work. Is there something that is not well configured? Can anyone help in this? I want to know If I'm allowed to run several jobs to upload the data using the same agent from my data center to s3

Answer

Hi!

Remember the following about running multiple tasks:

1. Tasks run in a series (first in, first out). If you run the same task more than once, by default each execution goes into a queue. If you start a task twice, for instance, the second task execution won't start until the first one finishes.
3. You can run separate tasks at the same time, but these tasks will be queued if they're using the same DataSync agent.

You can also take a look at this document: https://aws.amazon.com/es/blogs/storage/how-to-accelerate-your-data-transfers-with-aws-datasync-scale-out-architectures/

You may need to  set multiple agents to run the same task in  parallel 
![Enter image description here](/media/postImages/original/IMaoZw_feRTeSdeX_BRAIGsw)

Run multiple Datasync jobs in parallel

Relevant content