Multiple Lambda Execution for one EventsBridge Trigger

Question

Hi,

I am using AWS Lambda to write data every 5-mins to a bigquery table and google sheets and the bigquery table is linked to a near real time chart. This is working fine in most cases but in some instances it appears that AWS Lambda is getting executed multiple times and writes data for same batch time twice or thrice.  I could link this to Lambda execution time plot from Cloudwatch - whenever there is spike in execution time, I am getting duplicate data set in my big query. Though the plot says single invocation, there appears to be multiple invocation as I see some changes in values of some fields that I am measuring even though the batch time and associated primary keys for a measure are same.

Below is the python code snippet for the data write operation:
`bq.writeToBigQueryTable('my-project', 'my-dataset', 'my-table', mydataframe)
updateGoogleSheet("my-sheet-id",mylist) #my function which calls Google Sheets API`

Here is the graph for Lambda from cloudwatch: https://ibb.co/gWsRtfN

Would be grateful on any pointers on how to fix this.

Regards,
dbeings

Accepted Answer

The init phase is limited to 10 seconds, although, after the first 10 seconds we should retry it. The init phase is anything happening outsize the handler. Are you doing anything intensive in there? If you do, I suggest you move it into the handler.

Answer

You did not mention how you trigger the function. For example, if you trigger it from SQS, it may be that your visibility timeout is too low, so the function may be triggered more than once. Also, SQS has an At Least once lambda invocation, so by design your function may be invoked twice.

If idempotancy is important for your use case, you may need to implement it into your function by marking tha batch when it is being processed. You can mark it in a DynamoDB table for instancee

Multiple Lambda Execution for one EventsBridge Trigger

Contenido relevante