Batch download files from multiple different folders in the same S3 bucket
I have a csv file with a column specifying the names of the files that I would like to download. These files are stored in multiple different folders in the same S3 buckets. Is there a way I could download these files by passing the file names (either in a text file or a csv column), have AWS search in all the folders inside this S3 bucket, and download those files needed? I also have configured AWS CLI already if there's an option there. Thanks!
Here is a link to download file from S3 using python. https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3-example-download-file.html Try it out and see if you are able to download single files. You can then pass a text file to python program and for each filename in the text file, run the above code to download the file. Hope it helps
Relevant questions
unable to access S3 log files owned by "s3-log-service"
asked 2 years agoMove files from S3 Bucket back to Site
asked 2 months agoGlue ETL job write part-r-00 files to same bucket as my input. Any way to change this?
Accepted Answerasked 2 months agos3 Glacier Transition
asked 3 months agoBatch download files from multiple different folders in the same S3 bucket
asked 3 months agoDownloads Files From Workdocs
asked 3 months agoCan't download file from S3 bucket in another account
asked 2 years agoCan't get Partitions to work with my Glue Data Catalog
Accepted Answerasked 4 days agoS3 Static Website Objects 403 Forbidden when Uploaded from Different Account
asked a month agoCSV crawler tables name
asked 4 months ago
Thank you for your answer, Sandeep. I tried specifying my bucket, key, and filename just as what is showing in this example - s3.Bucket('mybucket').download_file('hello.txt', '/tmp/hello.txt'), but I kept running into this - ClientError: An error occurred (404) when calling the HeadObject operation: Not Found. So the 'hello.txt' in their example is a single file, stored in the folder 'tmp' inside the bucket called 'mybucket', correct? Just trying to understand the file structure here.