Amazon s3 filtering batch of object from a bucket by content

0

Im trying to build a datagrid page with objects from amazon s3, with functions like: filtering and pagination. I tried to filter data by values from the object and implement the limit and skip functionality( take n objects after m filtered objects). How can i implement this easily? Without fetching all the keys and values. (I tried to fetch all the objects and use s3 select to not load them in the memory, but i got an xml with payload failure) I use .NET btw

gandolh
asked 2 years ago424 views
1 Answer
0

The best way to achieve this use case is to implement a secondary indexing database in DynamoDB for the prefix and object names that reside in your S3 bucket. S3 is not designed to be able to perform server-side filtering for LIST API calls. Additionally, LIST API calls can become very expensive over time on Amazon S3. DynamoDB would be a much more suitable for this use case as you would be able to query the data server-side and only returning the required results to the client.

The following AWS Blog post dicusses how you can build a secondary indexing database for Amazon S3 on DynamoDB: https://aws.amazon.com/blogs/big-data/building-and-maintaining-an-amazon-s3-metadata-index-without-servers/

Additionally, you can also use S3 Inventory Reports to get a list of objects in your Amazon S3 bucket, without having to setup a secondary indexing database on DynamoDB: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

profile pictureAWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions