AWS Datasync when executing a task and not finding changes to make, counts and charges Requests on an S3 Bucket when performing a verified on the directory / where N files are located ?????


Dear, a task was created in Datasync which transferred 8 files within a directory and subdirectories. My query is when reviewing the logs of the Datasync task, I observe that different requests are generated (created, transferred, verified). When executing a new Datasync task where no transfer was made because no new changes were detected, but when reviewing the task log again, I observed that the request was made on the root (verified directory /). My question is, when Datasync does not find a change to make, it makes N requests on the total number of files and directories that are already updated. Example: if I have 15,000 files updated on a bucket, and when executing a task again that does not make changes, will AWS still charge me and count for listing all the files that have already been transferred? I attach images where I only have 22 objects including directories, files and subdirectories, where 43 tasks were executed, of which 40 did not transfer files (only verified directory /), which, checking my cost manager, made about 1840 Requests (Put, Copy ,Post or List Request) to Amazon S3 Enter image description here
Enter image description here
Enter image description here

1 Answer


When DataSync is targeting an S3 source or destination location the service does need to make API calls such a LIST to scan for changes to objects in the bucket. These API calls are part of request charges to S3 as outlined by storage class.

Even when there are no files transferred the DataSync task needs to list the contents of the bucket to inventory and determine what objects need to be transferred. This is noted under additional charges here:

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions