Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Lambda Function Not Executing on 16th Retry with SQS - Need Help with Long Batch Processes

0

Hello AWS Community,

I'm facing a challenge with an AWS Lambda function that is supposed to handle long-running batch processes. Despite setting up my system to delete and re-send SQS messages within the code to accommodate long processing times, the Lambda function fails to execute on the 16th retry. Each message is successfully re-queued with a new message ID, indicating that the re-sending part works as expected.

Here are some specifics:

  • I delete the SQS message being processed and re-send it as a new message to allow for extended processing time beyond the visibility timeout.
  • The "Maximum Receives" setting for the SQS queue is set to 2, which seems unrelated to the issue at hand, as each re-sent message is treated as new.
  • There are no indications of errors in the logs that would explain why the Lambda function doesn’t trigger for the 16th retry.

Could there be a misunderstanding on my part regarding how AWS handles these re-sent messages, or is there a configuration oversight? Any insights into handling long batch processes with Lambda and SQS, or similar experiences, would be greatly appreciated!

Thank you for your assistance!

Topics
Application IntegrationServerlessCompute
Tags
AWS LambdaAmazon Simple Queue Service
Language
English

asked 2 years ago453 views

2 Answers
2
Accepted Answer

Hello.

I think a Lambda recurrence loop may be detected.
It's hard to imagine the detailed configuration from just the text, but if the configuration is like SQS1 → Lambda → SQS1, it will be detected as a loop and the process will stop.
https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html

If your function is invoked more than 16 times in the same chain of requests, then Lambda automatically stops the next function invocation in that request chain and notifies you. If your function is configured with multiple triggers, then invocations from other triggers aren't affected.

EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

EXPERT

reviewed 2 years ago

  • pompom
    2 years ago

    Thank you so much for your insightful response. It appears that you have identified the exact issue I am facing with the Lambda function recursion loop. The configuration you described (SQS1 → Lambda → SQS1) matches mine, and it makes perfect sense that AWS would detect this as a loop and stop the process to prevent recursion issues, as detailed in the documentation you linked.

    This was precisely the information I needed to understand the behavior I was observing. I will adjust my approach to avoid this recursion and reconfigure my setup accordingly.

    Thank you again for your help. Your guidance has been invaluable!

0

Not answering the question but have you considered using Step Functions in this scenario? It seems like you're doing a bit of extra work (in terms of handling SQS message sending and receiving) that you might not have to.

EXPERT

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content