S3 Lambda Function slower than event trigger

Question

Hi Everyone,  
  
  Is there a time wait or sleep when performing lambda function. By that I mean, your lambda trigger is on S3 object creation, so the lambda function will trigger the moment a single file is created, what when the lambda function has completed its execution another file has already been created?  
Or lamda function is executed on single file creation? meaning 10 simultaneous files created in s3 10 times lambda function will be invoked?  
  
Thanks  
Sid

Accepted Answer

If this is your entire code, you seem to be missing a handler function (https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html). Code at the top level in a lambda function only gets run once, on instance initialization, but the handler function is called each time. Additionally, you don't need to tag every object on every invocation, only the object mentioned in the event passed to the handler function.  
  
As a side note, when posting code here, it's best to surround it with {\code\}(omit the backslashes) tags so it keeps formatting, like this:

```
def myfunc():
    return 'foo'
```

Answer

A invocation occurs for every object uploaded to S3. If an instance is already processing an event, and a new event comes in, a new instance will be created to handle the new event.  
  
https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

Answer

Thanks @Ellison, However I am having some issue, when lambda function is invoked its able to tag only few objects which are created, rest of them aren't tagged, whereas in when I run the python code from an EC2 instance it works perfectly fine tagging all the objects, so code works it's just that lambda is unable to tag the files as fast as they are uploaded.  
  
Any thoughts?

Answer

Thanks Ellison, New to lambda functions, will make a point to post code the way you suggested.

Answer

Thanks @Ellison , so is there a limit to number of instances it will spin up for continuous S3 event?  
  
So if 10 objects are created by a file transfer tool and each object takes just 2 seconds to upload then 10 lambda functions will be invoked?

Answer

Could you post your code please?

Answer

It is possible that the lambda is exiting before the S3 tagging has completed, thus aborting the tagging. Review the following link to see if adding async/promise/callback makes sense in your scenario.  
Link:

Answer

Hi Ellison,  
  
  Below is my code.  
  
import boto3  
  
# Create an S3 client  
s3 = boto3.client("s3")  
  
all_objects = s3.list_objects(Bucket = 'Bucket-Name')  
  
tag={ 'TagSet1' : {  'FileType1': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType2': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType3': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } }, 'TagSet2' : {  'FileType4': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType5': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType6': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } } }  
  
def tag_function(Folder,Object,tag):  
	tag_set = \[]  
	for key, value in tag\[Folder]\[Object].items():  
		tagging = {}  
		tagging.update({'Key' : key})  
		tagging.update({'Value' : value})  
		tag_set.append(tagging)  
	return tag_set  
  
for obj in all_objects\['Contents']:  
    object_path=obj\['Key']  
    try:  
        remove_extn=(object_path.split('.')\[0])  
        if remove_extn:  
            file=(remove_extn.split('/')\[-1])  
            if file:  
                Folder=(file.split('_')\[0])  
                Object=(file.split('_')\[1])  
                if Folder in \['TagSet1', 'TagSet2']:  
                    response = s3.put_object_tagging(  
                        Bucket='Bucket-Name',  
                        Key=object_path,  
                        Tagging={  
                            'TagSet': tag_function(Folder,Object,tag)  
                        }  
                    )  
    except IndexError:  
        pass

Answer

The default limit is 1,000 concurrent invocations, though this can be raised by submitting a ticket to support. https://docs.aws.amazon.com/lambda/latest/dg/limits.html  
  
Assuming that the uploads for all 10 objects completed at about the same time, 10 lambda instances will be created to handle them. But it can vary a little depending on timing.

S3 Lambda Function slower than event trigger

Create an S3 client

Relevant content