As mentioned in the Boto3 documentation that API will only return 1,000 objects. At that point you have to call again using the
ContinuationToken parameter using the
NextContinuationToken value received in the response.
Another option would be to use a paginator which makes your code a little simpler.
That said: Listing the contents of the bucket to find the latest object is not the most efficient way of doing that especially if there are many objects in the bucket. If it were me (and this depends highly on whether you just want the latest object or the last (say) ten latest objects): I would have a trigger for new objects in S3 that go to a Lambda function that stores the latest object into a DynamoDB table. Then you only have to query the table to get the latest object.
If you needed the latest 10 objects then the logic is a little more complex in the Lambda function but not particularly so. You might need to be careful if there are multiple uploads to S3 at the same time.
Thanks a lot Brettski-AWS. For now I have deleted the old objects and its working fine. In sometime, I will refactor the code to pull by the file name - need to standardize the generated file name by <prefix>_batch_time rather than <prefix>_update_time. But just two questions:
- In the sort objects, is there some way in which sort runs on entire bucket on last modified timestamp and the query returns just the first 1000. As in some approach where object fetch could be paginated but sort work on entire bucket?
- Just curious, why did it break only post a particular time each day.
Instead of listing all files, you could use S3 Inventory: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket.
- Accepted Answerasked 3 days ago
- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated 4 months ago
- How do I set up Amazon S3 event notifications to invoke a Lambda function that's in another AWS account?AWS OFFICIALUpdated a year ago
- EXPERTpublished a year ago