S3 Lambda Function slower than event trigger

0

Hi Everyone,

Is there a time wait or sleep when performing lambda function. By that I mean, your lambda trigger is on S3 object creation, so the lambda function will trigger the moment a single file is created, what when the lambda function has completed its execution another file has already been created?
Or lamda function is executed on single file creation? meaning 10 simultaneous files created in s3 10 times lambda function will be invoked?

Thanks
Sid

sid1987
asked 3 years ago165 views
9 Answers
0
Accepted Answer

If this is your entire code, you seem to be missing a handler function (https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html). Code at the top level in a lambda function only gets run once, on instance initialization, but the handler function is called each time. Additionally, you don't need to tag every object on every invocation, only the object mentioned in the event passed to the handler function.

As a side note, when posting code here, it's best to surround it with {\code}(omit the backslashes) tags so it keeps formatting, like this:

def myfunc():
    return 'foo'
Ellison
answered 3 years ago
0

A invocation occurs for every object uploaded to S3. If an instance is already processing an event, and a new event comes in, a new instance will be created to handle the new event.

https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

Ellison
answered 3 years ago
0

Thanks @Ellison , so is there a limit to number of instances it will spin up for continuous S3 event?

So if 10 objects are created by a file transfer tool and each object takes just 2 seconds to upload then 10 lambda functions will be invoked?

sid1987
answered 3 years ago
0

The default limit is 1,000 concurrent invocations, though this can be raised by submitting a ticket to support. https://docs.aws.amazon.com/lambda/latest/dg/limits.html

Assuming that the uploads for all 10 objects completed at about the same time, 10 lambda instances will be created to handle them. But it can vary a little depending on timing.

Ellison
answered 3 years ago
0

Thanks @Ellison, However I am having some issue, when lambda function is invoked its able to tag only few objects which are created, rest of them aren't tagged, whereas in when I run the python code from an EC2 instance it works perfectly fine tagging all the objects, so code works it's just that lambda is unable to tag the files as fast as they are uploaded.

Any thoughts?

sid1987
answered 3 years ago
0

Could you post your code please?

Ellison
answered 3 years ago
0

It is possible that the lambda is exiting before the S3 tagging has completed, thus aborting the tagging. Review the following link to see if adding async/promise/callback makes sense in your scenario.
Link: https://stackoverflow.com/questions/55012396/write-to-s3-bucket-using-async-await-in-aws-lambda

answered 3 years ago
0

Hi Ellison,

Below is my code.

import boto3

Create an S3 client

s3 = boto3.client("s3")

all_objects = s3.list_objects(Bucket = 'Bucket-Name')

tag={ 'TagSet1' : { 'FileType1': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType2': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType3': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } }, 'TagSet2' : { 'FileType4': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType5': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType6': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } } }

def tag_function(Folder,Object,tag):
tag_set = []
for key, value in tag[Folder][Object].items():
tagging = {}
tagging.update({'Key' : key})
tagging.update({'Value' : value})
tag_set.append(tagging)
return tag_set

for obj in all_objects['Contents']:
object_path=obj['Key']
try:
remove_extn=(object_path.split('.')[0])
if remove_extn:
file=(remove_extn.split('/')[-1])
if file:
Folder=(file.split('')[0])
Object=(file.split('
')[1])
if Folder in ['TagSet1', 'TagSet2']:
response = s3.put_object_tagging(
Bucket='Bucket-Name',
Key=object_path,
Tagging={
'TagSet': tag_function(Folder,Object,tag)
}
)
except IndexError:
pass

sid1987
answered 3 years ago
0

Thanks Ellison, New to lambda functions, will make a point to post code the way you suggested.

sid1987
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions