bulk move files from root of an S3 bucket to a folder (but not include other folders)


Seems like a simple request, yet having some troubles. Basically I have a lot of folders of client info, under each client is a lot of files (some over 500,000 files), so moving forward I am now breaking things out by time. So I am having a simple yyyy-mm setup and its been running great, so for cleanup I have an archive folder and want to move all files in the root of each client (but of course not 2021-12, 2022-01 folder contents in) to that archive folder.

So, in a normal environment, I could goto that root, move * archive/ and done. But with them in S3, it's a bit tougher. I am testing, but I don't see how to get it. Of course my thought was; aws s3 mv s3://bucket/folder1/ s3://bucket/folder/archive* but naturally no good. The error is quite odd; fatal error: An error occurred (404) when calling the HeadObject operation: Key "documents/client1/" does not exist (but of course it does), if I add an --recursive, it does work (but does the other folders)

I see there is an include, so thought maybe I could find a way to throw in an --exclude 2021-12/ 2022-01

Any help is appreciated.

2 Answers
Accepted Answer

From the CLI consider using --recursive --exclude "*/*" this should prevent copying of any objects with a / in the key which you can use to indicate a sub-folder. Be aware that this will process every object in the source location as it will need to determine if the key contains the exclude pattern.

Further info at Use of Exclude and Include Filters

answered a year ago
  • Thanks @RoB - unfortunately, didn't work. I didn't get an error, however it did move the root folders;

    command: aws s3 mv s3://bucket/documents/00000154/ s3://bucket/documents/00000154/archive/ —recursive —exclude “/”

    result: move: s3://bucket/documents/00000154/Index/_2j.fdx to s3://bucket/documents/00000154/archive/Index/_2j.fdx

    Naturally there were hundreds of the root that did move, but I had 2 folders in the root that got moved into the archive folder.


@Rob - after seeing it not work, I tried a few --exclude combinations and got it, then re-visited the post/my command and neglected the *, so re-tested and worked perfect!

Thanks much, this was an easy / yet annoying one!

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions