S3 Lambda Function slower than event trigger

0

Hi Everyone,

Is there a time wait or sleep when performing lambda function. By that I mean, your lambda trigger is on S3 object creation, so the lambda function will trigger the moment a single file is created, what when the lambda function has completed its execution another file has already been created?
Or lamda function is executed on single file creation? meaning 10 simultaneous files created in s3 10 times lambda function will be invoked?

Thanks
Sid

sid1987
已提问 5 年前798 查看次数
9 回答
0
已接受的回答

If this is your entire code, you seem to be missing a handler function (https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html). Code at the top level in a lambda function only gets run once, on instance initialization, but the handler function is called each time. Additionally, you don't need to tag every object on every invocation, only the object mentioned in the event passed to the handler function.

As a side note, when posting code here, it's best to surround it with {\code}(omit the backslashes) tags so it keeps formatting, like this:

def myfunc():
    return 'foo'
Ellison
已回答 5 年前
profile picture
专家
已审核 9 个月前
0

A invocation occurs for every object uploaded to S3. If an instance is already processing an event, and a new event comes in, a new instance will be created to handle the new event.

https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

Ellison
已回答 5 年前
profile picture
专家
已审核 9 个月前
0

Thanks @Ellison , so is there a limit to number of instances it will spin up for continuous S3 event?

So if 10 objects are created by a file transfer tool and each object takes just 2 seconds to upload then 10 lambda functions will be invoked?

sid1987
已回答 5 年前
0

The default limit is 1,000 concurrent invocations, though this can be raised by submitting a ticket to support. https://docs.aws.amazon.com/lambda/latest/dg/limits.html

Assuming that the uploads for all 10 objects completed at about the same time, 10 lambda instances will be created to handle them. But it can vary a little depending on timing.

Ellison
已回答 5 年前
0

Thanks @Ellison, However I am having some issue, when lambda function is invoked its able to tag only few objects which are created, rest of them aren't tagged, whereas in when I run the python code from an EC2 instance it works perfectly fine tagging all the objects, so code works it's just that lambda is unable to tag the files as fast as they are uploaded.

Any thoughts?

sid1987
已回答 5 年前
0

Could you post your code please?

Ellison
已回答 5 年前
0

It is possible that the lambda is exiting before the S3 tagging has completed, thus aborting the tagging. Review the following link to see if adding async/promise/callback makes sense in your scenario.
Link: https://stackoverflow.com/questions/55012396/write-to-s3-bucket-using-async-await-in-aws-lambda

已回答 5 年前
0

Hi Ellison,

Below is my code.

import boto3

Create an S3 client

s3 = boto3.client("s3")

all_objects = s3.list_objects(Bucket = 'Bucket-Name')

tag={ 'TagSet1' : { 'FileType1': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType2': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType3': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } }, 'TagSet2' : { 'FileType4': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType5': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } , 'FileType6': { 'Key1': 'Value1', 'Key2': 'Value2', 'Key3': 'Value3' } } }

def tag_function(Folder,Object,tag):
tag_set = []
for key, value in tag[Folder][Object].items():
tagging = {}
tagging.update({'Key' : key})
tagging.update({'Value' : value})
tag_set.append(tagging)
return tag_set

for obj in all_objects['Contents']:
object_path=obj['Key']
try:
remove_extn=(object_path.split('.')[0])
if remove_extn:
file=(remove_extn.split('/')[-1])
if file:
Folder=(file.split('')[0])
Object=(file.split('
')[1])
if Folder in ['TagSet1', 'TagSet2']:
response = s3.put_object_tagging(
Bucket='Bucket-Name',
Key=object_path,
Tagging={
'TagSet': tag_function(Folder,Object,tag)
}
)
except IndexError:
pass

sid1987
已回答 5 年前
0

Thanks Ellison, New to lambda functions, will make a point to post code the way you suggested.

sid1987
已回答 5 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则