sqs event triggers lambda directly, is there a way to delay that execution by 10-20 seconds?


I am building a serverless scraper where I have one lambda function that enqueues n urls of m websites (n>>m) one by one into a message queue. Then I have a sqs trigger that invokes another lambda which takes the url and scrapes it.

I set the batch size of the sqs trigger to 1 and the limited the lambda to 1 concurrent invocation. The problem is, that I want to have a pause of 20-40 seconds between each scraping, but currently it applies no pause and only executes the function(takes about 10 seconds).

Is there a way to make that work or do I have to rethink my current architecture?

Update: So the problem is, that adding a delay to the messages does not solve my problem. When I use the delay, x seconds are added till each message is dequeued. But as all 3 seconds a new messages are added, the dequeue operation is just 20 seconds later so the lambda is executed at 20, 23, 26, 29, 32 instead of 20, 40, 60, 80, 100 seconds after initial enqueue.

Solution: I took a modified approach. I now have a cron event, that triggers a lambda each minute. The lambda then gets the message from the queue. The modification is, that the cron job gets disabled when the queue is empty. If elements are added it is enabled again etc. Thank you all!

1 Answer
Accepted Answer

You could also have a lambda that is triggered by a timer (cloudwatch event) every 30s. It would read one item from the queue or exit immediately if there are no messages on the queue. One invocation every 30s is fairly cheap, and you are guaranteed to have no more than that rate of scrapes.

profile picture
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions