- Newest
- Most votes
- Most comments
Hello.
I also agree with that opinion.
In the system I created in the past, I recorded the object path in DynamoDB when saving images to S3, and referenced the object path from DynamoDB when downloading the object.
https://aws.amazon.com/jp/blogs/big-data/building-and-maintaining-an-amazon-s3-metadata-index-without-servers/
Take a look at S3 Inventory
https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html
Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing.
I also agree with earlier remarks, but I'd like to emphasise that particularly with DynamoDB, you should first clearly define your access patterns to the data and design the key contents and storage formats based on them. That's what's explained in detail in that blog post for a certain scenario, with data arranged based on customer etc.
The specifics of your environment are likely to differ. In any case, simply dumping an object inventory in DynamoDB and starting to scan it shouldn't be expected to perform well or cost-efficiently. If you want a quick-and-dirty solution, a SQL database, such as Aurora Serverless with PostgreSQL compatibility, will go a long way with little to no planning and with indexing, statistics, etc. jiggled into place later, while with DynamoDB, that approach won't generally work at all. With DynamoDB, the first step would be to model your access patterns, leading to the proper structures for your data.
Relevant content
- AWS OFFICIALUpdated a year ago
Note that this wouldn't be real-time data but, as described, a scheduled background task that only starts to generate the inventory at most once per day, at midnight UTC (regardless of your time zone), and take a variable amount of time to complete. It isn't useful in isolation for reflecting the current contents of a bucket undergoing continuous change.