We have a service that can take in list of destination S3 locations and a presigned URL of an assets from the source bucket as an input. The service downloads the asset from the presigned url and uploads it in the destination bucket with the S3 location provided.
To add more context, the assets are in sizes ~10GB and there can be 10000 requests per day. We would like to know if AWS datasync can be used to replace the current service? From the documentation, we found that for each transfer we would need to create source and destination locations, tasks to execute the transfer process.
If AWS datasync can be used in this scenario, are there any limitations on the number of clients we can onboard? This is because, AWSDatasync works with IAM polcies and if multiple clients onboard, we may have a limit on the size of the policy document. Are there any workarounds or limitations on using AWS DataSync?
AWS OFFICIALUpdated 3 years ago
