sqs event triggers lambda directly, is there a way to delay that execution by 10-20 seconds?

0

I am building a serverless scraper where I have one lambda function that enqueues n urls of m websites (n>>m) one by one into a message queue. Then I have a sqs trigger that invokes another lambda which takes the url and scrapes it.

I set the batch size of the sqs trigger to 1 and the limited the lambda to 1 concurrent invocation. The problem is, that I want to have a pause of 20-40 seconds between each scraping, but currently it applies no pause and only executes the function(takes about 10 seconds).

Is there a way to make that work or do I have to rethink my current architecture?


Update: So the problem is, that adding a delay to the messages does not solve my problem. When I use the delay, x seconds are added till each message is dequeued. But as all 3 seconds a new messages are added, the dequeue operation is just 20 seconds later so the lambda is executed at 20, 23, 26, 29, 32 instead of 20, 40, 60, 80, 100 seconds after initial enqueue.


Solution: I took a modified approach. I now have a cron event, that triggers a lambda each minute. The lambda then gets the message from the queue. The modification is, that the cron job gets disabled when the queue is empty. If elements are added it is enabled again etc. Thank you all!

1回答
0
承認された回答

You could also have a lambda that is triggered by a timer (cloudwatch event) every 30s. It would read one item from the queue or exit immediately if there are no messages on the queue. One invocation every 30s is fairly cheap, and you are guaranteed to have no more than that rate of scrapes.

profile pictureAWS
回答済み 4年前
profile picture
エキスパート
レビュー済み 10ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ