bulk move files from root of an S3 bucket to a folder (but not include other folders)

Question

Seems like a simple request, yet having some troubles.   Basically I have a lot of folders of client info, under each client is a lot of files (some over 500,000 files), so moving forward I am now breaking things out by time.  So I am having a simple yyyy-mm setup and its been running great, so for cleanup I have an archive folder and want to move all files in the root of each client (but of course not 2021-12, 2022-01 folder contents in) to that archive folder.

So, in a normal environment, I could goto that root, move * archive/ and done.   But with them in S3, it's a bit tougher.   I am testing, but I don't see how to get it.  Of course my thought was;
**aws s3 mv s3://bucket/folder1/* s3://bucket/folder/archive**  but naturally no good. The error is quite odd;
**fatal error: An error occurred (404) when calling the HeadObject operation: Key "documents/client1/" does not exist** (but of course it does), if I add an --recursive, it does work (but does the other folders)

I see there is an include, so thought maybe I could find a way to throw in an --exclude 2021-12/ 2022-01

Any help is appreciated.

RoB · Accepted Answer

From the CLI consider using **--recursive --exclude "\*/\*"** this should prevent copying of any objects with a / in the key which you can use to indicate a sub-folder. Be aware that this will process every object in the source location as it will need to determine if the key contains the exclude pattern.

Further info at [Use of Exclude and Include Filters](https://docs.aws.amazon.com/cli/latest/reference/s3/index.html#use-of-exclude-and-include-filters)

lraymond · Answer

@Rob - after seeing it not work, I tried a few --exclude combinations and got it, then re-visited the post/my command and neglected the *, so re-tested and worked perfect!

Thanks much, this was an easy / yet annoying one!

bulk move files from root of an S3 bucket to a folder (but not include other folders)

Relevant content