Issue in accessing latest generated file in S3 bucket

Question

Hi,

I am generating a file in S3 every hour through a Lambda function and then after 10 minutes call a second lambda function processLatestS3File(). The second lambda function is called 10 minutes after the first one and the first one generates the file within a minute.

This code works as expected till about 1030 AM UTC each day and the second lambda function gets reference to the latest generated file. But post that, in subsequent calls for the day the second lambda function is always getting the 1030 AM UTC file when it tries to get the latest file in the bucket.

This was working perfectly fine for entire day few weeks back and I have made no changes to code. I have also checked and the first lambda function is working fine and is generating required files every hour even after 1030 AM UTC. I also checked the modified timestamp in S3 bucket and files generated after 1030 AM have more recent modified timestamps than the 1030 AM UTC file. Below is the code extract in processLatestS3File() to get the latest update file in bucket.

```
response = s3.list_objects_v2(Bucket=my_bucket_name)
sorted_objects = sorted(response['Contents'], key=lambda obj: obj['LastModified'], reverse=True)
latest_object = sorted_objects[0]
print(sorted_objects1[0]['Key'])
      
```

Any pointers on what could be causing this?

Regards,
dbeings

Accepted Answer

As mentioned in [the Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_objects_v2.html) that API will only return 1,000 objects. At that point you have to call again using the `ContinuationToken` parameter using the `NextContinuationToken` value received in the response.

Another option would be to use [a paginator](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html) which makes your code a little simpler.

That said: Listing the contents of the bucket to find the latest object is not the most efficient way of doing that especially if there are many objects in the bucket. If it were me (and this depends highly on whether you just want the latest object or the last (say) ten latest objects): I would have a trigger for new objects in S3 that go to a Lambda function that stores the latest object into a DynamoDB table. Then you only have to query the table to get the latest object.

If you needed the latest 10 objects then the logic is a little more complex in the Lambda function but not particularly so. You might need to be careful if there are multiple uploads to S3 at the same time.

Answer

Thanks a lot Brettski-AWS. For now I have deleted the old objects and its working fine. In sometime, I will refactor the code to pull by the file name - need to standardize the generated file name by  _batch_time  rather than _update_time. But just two questions:

1. In the sort objects, is there some way in which sort runs on entire bucket on last modified timestamp and the query returns just the first 1000. As in some approach where object fetch could be paginated but sort work on entire bucket?  
2. Just curious, why did it break only post a particular time each day.

Regards,
dbeings

Answer

Instead of listing all files, you could use S3 Inventory: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket.

Issue in accessing latest generated file in S3 bucket

Relevant content