Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Subject: Inconsistent Behavior in AWS CLI sync --delete Command Across Different Clients

0

Subject: Inconsistent Behavior in AWS CLI sync --delete Command Across Different Clients

Hello

We have observed that client requests vary, even when using the same AWS CLI command. Please review the specification below.

What we are doing:

We are using the aws s3 sync --delete command to update objects at the destination. Both the source and destination contain the same object, but the source object has a newer "Last Modified" timestamp.

Client A

Endpoint: 3rd party (same as Client B) OS: macOS 14.4 (Build 23E5196e) AWS CLI version: aws-cli/2.11.1 Python/3.11.2 Darwin/23.4.0 exe/x86_64 prompt/off Client B

Endpoint: 3rd party (same as Client A) OS: Rocky Linux release 8.10 (Green Obsidian) AWS CLI version: aws-cli/2.11.1 Python/3.11.2 Linux/4.18.0-553.36.1.el8_10.x86_64 exe/x86_64.rocky.8 prompt/off Observed Symptoms

Client A compares the "Last Modified" value between source and destination and only uploads when the source is newer. Client B also compares the "Last Modified" value, but when executing the sync, it requests both upload and delete operations simultaneously. Please advise on why the aws s3 sync --delete command behaves differently between the two systems under the same conditions.

Topics
Storage
Tags
Amazon Simple Storage Service
Language
English

asked a year ago139 views

2 Answers
1

The behavior you're observing with the aws s3 sync --delete command across different clients is related to how the AWS CLI handles file comparisons during synchronization operations.

By default, the AWS CLI S3 sync command uses both file size and modification time to determine if a file needs to be updated. However, the comparison of modification times works in a specific way that can appear counterintuitive:

  1. If the destination file is newer than the source file, the sync command will download/upload the file. This happens because the command assumes that the source version is the "authoritative" version, and any destination file with a more recent timestamp might be out of sync.

  2. If the source file is newer than the destination file, the sync command typically won't transfer the file again, assuming the newer source version is already reflected in the destination.

The difference in behavior between your macOS and Rocky Linux clients could be due to several factors:

  • Different filesystem timestamp precision or handling between macOS and Linux
  • Variations in how the operating systems handle file metadata
  • Potential clock synchronization differences between the systems
  • Subtle differences in how the AWS CLI interacts with the underlying operating system

To make the behavior more consistent across platforms, you could try using additional flags with your sync command:

  • Use --exact-timestamps to only update files if the timestamps are different
  • Use --size-only to compare only the size of the files, ignoring timestamps completely

The --delete flag you're using removes files from the destination that don't exist in the source, but the decision about which files to upload/download is still governed by the timestamp and size comparison logic.

This behavior is by design to minimize unnecessary data transfer, but it may not always align with specific use cases across different operating systems.
Sources
aws s3 sync --- downloading older version from source [BUG?] | AWS re:Post
Is AWS CLI really synchronious? | AWS re:Post

answered a year ago

EXPERT

reviewed a year ago

0

You can run the following command to see what will happen when you run sync.

aws s3 sync source/ s3://destination-bucket/ --delete --exact-timestamps --dryrun

Also,

  • Confirm whether the metadata of the object has changed or not
  • Confirm that you are using same timestamp when calling to S3
EXPERT

answered a year ago

  • ymk
    a year ago

    intention update object on destination

    1. The endpoint(3rd party) only supports comparing Content-Length and Last-Modified when executing sync --delete.
    2. Comparing ETag hashes is not supported.
    3. excuting same command.
    4. source station metadata does not match with destination object.

    A client (dryrun) upload: ./sync_1.jpg to s3://......../sync_1.jpg (dryrun) upload: ./sync_2.jpg to s3://......../sync_2.jpg (dryrun) upload: ./sync_sakje1.jpg to s3://........./sync_sakje1.jpg (dryrun) upload: ./sync_sakje2.jpg to s3://......../sync_sakje2.jpg (dryrun) upload: ./sync_sakje3.jpg to s3://......./sync_sakje3.jpg

    B client however B client sync --delete excuting both upload and delete operations simultaneously when source station meta data modified

    (dryrun) upload: ../../../../../....../common.css to s3://......./common.css (dryrun) delete: s3://......./common.css

    (dryrun) upload: ../../../../../....../fonts/Pretendard-Black.subset.woff to s3://....../fonts/Pretendard-Black.subset.woff (dryrun) delete: s3://......./fonts/Pretendard-Black.subset.woff

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content