Amazon s3 filtering batch of object from a bucket by content

Question

Im trying to build a datagrid page with objects from amazon s3, with functions like: filtering and pagination. I tried to filter data by values from the object and implement the limit and skip functionality( take n objects after m filtered objects). 
How can i implement this easily? Without fetching all the keys and values.
(I tried to fetch all the objects and use s3 select to not load them in the memory, but i got an xml with payload failure)
I use .NET btw

Answer

The best way to achieve this use case is to implement a secondary indexing database in DynamoDB for the prefix and object names that reside in your S3 bucket. S3 is not designed to be able to perform server-side filtering for LIST API calls. Additionally, LIST API calls can become very expensive over time on Amazon S3. DynamoDB would be a much more suitable for this use case as you would be able to query the data server-side and only returning the required results to the client.

The following AWS Blog post dicusses how you can build a secondary indexing database for Amazon S3 on DynamoDB:
https://aws.amazon.com/blogs/big-data/building-and-maintaining-an-amazon-s3-metadata-index-without-servers/

Additionally, you can also use S3 Inventory Reports to get a list of objects in your Amazon S3 bucket, without having to setup a secondary indexing database on DynamoDB: 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

Amazon s3 filtering batch of object from a bucket by content

Relevanter Inhalt