How to get S3 files information optimally

Question

I need to get information like file name, file size, file type from hierarchical S3 folder structure which is organized like business/sub_business/date/files_with_name_of_originator . I need to get the info for one particular sub_business and within that for one particular type of file (type contained in file name) for a date range. 
One Option is to Query the sub_business for each request. Other option can be to capture S3 events when files come and store meta data in RDS. I get daily 2 million files.  Is there a better way?

Brettski-AWS · Answer

If the "turnover rate" of your files is quite rapid then what you're doing is probably best. You might find that using a different data store (such as DynamoDB) may be less expensive and more performant (because DynamoDB works well as a key/value store) but that may require too many changes to your application.

However, you might consider using S3 Inventory noting that it may take up to 48 hours for files to appear in the inventory - that might not be appropriate for your requirements.

However, you might consider using [S3 Inventory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html) noting that it may take up to 48 hours for files to appear in the inventory - that might not be appropriate for your requirements.

How to get S3 files information optimally

Relevant content