Amazon s3 filtering batch of object from a bucket by content


Im trying to build a datagrid page with objects from amazon s3, with functions like: filtering and pagination. I tried to filter data by values from the object and implement the limit and skip functionality( take n objects after m filtered objects). How can i implement this easily? Without fetching all the keys and values. (I tried to fetch all the objects and use s3 select to not load them in the memory, but i got an xml with payload failure) I use .NET btw

asked 2 years ago424 views
1 Answer

The best way to achieve this use case is to implement a secondary indexing database in DynamoDB for the prefix and object names that reside in your S3 bucket. S3 is not designed to be able to perform server-side filtering for LIST API calls. Additionally, LIST API calls can become very expensive over time on Amazon S3. DynamoDB would be a much more suitable for this use case as you would be able to query the data server-side and only returning the required results to the client.

The following AWS Blog post dicusses how you can build a secondary indexing database for Amazon S3 on DynamoDB:

Additionally, you can also use S3 Inventory Reports to get a list of objects in your Amazon S3 bucket, without having to setup a secondary indexing database on DynamoDB:

profile pictureAWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions