Fetching S3 objects filtered by date

Question

I have a requirement for a python script running periodically, which checks how many objects were added to a specific S3 bucket during the last 24h. At the moment, the script has to fetch all objects from the bucket filtered by the prefix, and then filter them by date on the client side. Unfortunately, this takes over an hour, simply because we have so many objects in the bucket. I would like to find out if it's possible to fetch from S3 bucket only the objects from last 24h, which would massively speed up the script. I did not find such an option for the aws cli command, which means that the boto3 framework I use won't have it either. I have been looking into S3 inventory too, but it's same in this case - only prefix can be used to filter the objects, not a date. In my case, I cannot change naming format for the objects, and at the moment they don't include date. Is there I way of doing this that I missed?

Answer

Hi,

I would suggest a different approach: implement a Lambda trigger on S3 updates. and use this Lambda to record the changes in a S3 file. So, you'll just have to record the changes in a file from this Lambda. Then, downloading this single inventory file will give you all changes at once.

This doc gives you details about S3 triggers with Lamdda: https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

Best,

Didier

Answer

Hello.

If you cannot change object names, I don't think you can filter using S3's "ListObjectsV2" API.  
So, as an alternative, how about registering the object path and creation date in DynamoDB when the S3 object is created?  
By setting an S3 event trigger, you can run Lambda when an object is created.  
By using Lambda to register object information to DynamoDB, you can easily perform aggregation.  
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

Fetching S3 objects filtered by date

Relevant content