How to copy between S3 locations without iterating.

Question

I want to copy or move file from one s3 location to other within the same bucket( rename the prefix) in bulk. Currently I’m iterating over the summary object and copying with each key. 
I was able to delete files in bulk by providing the prefix which I want to delete. I would like to do the same with copy. What are my options here. I need a python implementation.
Using aws cli via subprocess doesn’t seem right because I’m not getting any response if copy is success/failure (I guess).
And s3 batch jobs also doesn’t seems to be right as it is preferred for large no. files and pricing as well.

The ultimate goal is to reduce the no. boto3 calls from my program. I need a way to just give the source and destination prefix and move the files from source to destination.

Answer

Hello.

In the case of boto3, I don't think it is possible to copy all at once.  
I think it would be better to create a Python script that retrieves the copy source objects all at once using s3.list_objects_v2() and executes copy_object() for each object.  
For example, the code below behaves like copying objects one by one.  
```
import boto3

next_token = ''
source_prefix = 'source-folder/'
target_prefix = 'target-folder/'
s3 = boto3.client('s3')

while True:
    if next_token == '':
        response = s3.list_objects_v2(Bucket="S3-Source-Bucket-Name", Prefix=source_prefix)
    else:
        response = s3.list_objects_v2(Bucket="S3-Source-Bucket-Name", Prefix=source_prefix, ContinuationToken=next_token)
    
    if 'Contents' in response:
        contents = response['Contents']
        for content in contents:
            s3.copy_object(Bucket="S3-Target-Bucket-Name", Key=target_prefix, CopySource={'Bucket': "S3-Source-Bucket-Name", 'Key': content['Key']})

if 'NextContinuationToken' in response:
        next_token = response['NextContinuationToken']
    else:
        break
```

How to copy between S3 locations without iterating.

Relevant content