By using AWS re:Post, you agree to the Terms of Use
/aws s3 sync syncstrategy shows incorrect timestamp/

aws s3 sync syncstrategy shows incorrect timestamp

0

My typical use is to sync a series of directories/subdirectories from S3 to a cifs mounted SMB share on a local Linux machine. After a recent local server reboot/remount of network storage, the sync command now re-transfers EVERYTHING in the directories every time I run it. My belief is that the sync command is pulling the CURRENT time as the local timestamp instead of the modify time of the files. I ran it with dryrun and debug, and got a series of syncstrategy statements that appear to show a comparison between the S3 and local file. If I'm reading this correctly, the filesize is the same, the S3 timestamps showing correctly, but the local file timestamp is showing the immediate current time. The local linux environment ls shows the correct modified timestamp, which matches the s3 ls of the same file.

Here is example output from the debug. Note the "modify time:" section. I believe that this shows the correct modify time for the S3 files, but shows the time the command was run for the local files. (modified time: 2022-03-18 16:52:02-07:00 -> 2022-03-24 12:48:39.973111-07:00 <-- the command was run at this datetime, and it seems to climb with each file as the seconds tick by)

2022-03-24 12:48:40,066 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-10.mp3 -> /mnt/qnapprd_integration/AOD/FZAOD/album-VariousArtists_TimelessHits-10.mp3, size: 2827911 -> 2827911, modified time: 2022-03-18 16:52:02-07:00 -> 2022-03-24 12:48:39.973111-07:00
(dryrun) download: s3://com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-10.mp3 to FZAOD/album-VariousArtists_TimelessHits-10.mp3
2022-03-24 12:48:40,066 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-11.mp3 -> /mnt/qnapprd_integration/AOD/FZAOD/album-VariousArtists_TimelessHits-11.mp3, size: 3248378 -> 3248378, modified time: 2022-03-18 16:52:12-07:00 -> 2022-03-24 12:48:39.945111-07:00
(dryrun) download: s3://com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-11.mp3 to FZAOD/album-VariousArtists_TimelessHits-11.mp3
2022-03-24 12:48:40,067 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-12.mp3 -> /mnt/qnapprd_integration/AOD/FZAOD/album-VariousArtists_TimelessHits-12.mp3, size: 4518138 -> 4518138, modified time: 2022-03-18 16:52:12-07:00 -> 2022-03-24 12:48:39.981111-07:00
(dryrun) download: s3://com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-12.mp3 to FZAOD/album-VariousArtists_TimelessHits-12.mp3
2022-03-24 12:48:40,067 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-13.mp3 -> /mnt/qnapprd_integration/AOD/FZAOD/album-VariousArtists_TimelessHits-13.mp3, size: 8270994 -> 8270994, modified time: 2022-03-18 16:53:03-07:00 -> 2022-03-24 12:48:40.001111-07:00
(dryrun) download: s3://com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-13.mp3 to FZAOD/album-VariousArtists_TimelessHits-13.mp3
2022-03-24 12:48:40,068 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: com.my.bucket.share/FZAOD/album-VariousArtists_TimelessHits-14.mp3 -> /mnt/qnapprd_integration/AOD/FZAOD/album-VariousArtists_TimelessHits-14.mp3, size: 5135882 -> 5135882, modified time: 2022-03-18 16:52:33-07:00 -> 2022-03-24 12:48:39.941111-07:00

Does anyone have any insight into how this timestamp is pulled, or what might stop s3 sync from retrieving the correct modify time of local files?

1 Answers
0
Accepted Answer

Turns out this was a problem with the CIFS mount itself. Although all seems intact and correct, something is happening at mount that causes the s3 sync to not be able to correctly access the files, or at least some part of their properties. An unmount/remount of the share seems to correct the problem.

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions