sqs event triggers lambda directly, is there a way to delay that execution by 10-20 seconds?

0

I am building a serverless scraper where I have one lambda function that enqueues n urls of m websites (n>>m) one by one into a message queue. Then I have a sqs trigger that invokes another lambda which takes the url and scrapes it.

I set the batch size of the sqs trigger to 1 and the limited the lambda to 1 concurrent invocation. The problem is, that I want to have a pause of 20-40 seconds between each scraping, but currently it applies no pause and only executes the function(takes about 10 seconds).

Is there a way to make that work or do I have to rethink my current architecture?


Update: So the problem is, that adding a delay to the messages does not solve my problem. When I use the delay, x seconds are added till each message is dequeued. But as all 3 seconds a new messages are added, the dequeue operation is just 20 seconds later so the lambda is executed at 20, 23, 26, 29, 32 instead of 20, 40, 60, 80, 100 seconds after initial enqueue.


Solution: I took a modified approach. I now have a cron event, that triggers a lambda each minute. The lambda then gets the message from the queue. The modification is, that the cron job gets disabled when the queue is empty. If elements are added it is enabled again etc. Thank you all!

已提問 4 年前檢視次數 2585 次
1 個回答
0
已接受的答案

You could also have a lambda that is triggered by a timer (cloudwatch event) every 30s. It would read one item from the queue or exit immediately if there are no messages on the queue. One invocation every 30s is fairly cheap, and you are guaranteed to have no more than that rate of scrapes.

profile pictureAWS
已回答 4 年前
profile picture
專家
已審閱 9 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南