S3 query file name without database

Question

hi,
I'm have this question. I need to create a document portal to download product manuals. The number of files is very high, we're talking about approximately 200,000 files totaling around 100GB in size. The files are stored in an S3 bucket. My portal should display the download link for all files whose file name contains a serial number. i've multiple manual for every product that in the name contain serial number. I'd prefer not to use a supporting database. My application executes a LIST operation on the objects to search for it, which impacts the S3 LIST costs significantly.

What's the best approach in these cases? Please consider also that the files are updated six times a day.

Answer

Use presigned URLs, either from S3 directly (https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html) or with cloudfront (https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html).

For listing, if using Python you can try paginators: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html, otherwise leverage SDK and paging: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/the-response-object.html#response-paged-data

S3 query file name without database

Relevant content