sqs event triggers lambda directly, is there a way to delay that execution by 10-20 seconds?

0

I am building a serverless scraper where I have one lambda function that enqueues n urls of m websites (n>>m) one by one into a message queue. Then I have a sqs trigger that invokes another lambda which takes the url and scrapes it.

I set the batch size of the sqs trigger to 1 and the limited the lambda to 1 concurrent invocation. The problem is, that I want to have a pause of 20-40 seconds between each scraping, but currently it applies no pause and only executes the function(takes about 10 seconds).

Is there a way to make that work or do I have to rethink my current architecture?


Update: So the problem is, that adding a delay to the messages does not solve my problem. When I use the delay, x seconds are added till each message is dequeued. But as all 3 seconds a new messages are added, the dequeue operation is just 20 seconds later so the lambda is executed at 20, 23, 26, 29, 32 instead of 20, 40, 60, 80, 100 seconds after initial enqueue.


Solution: I took a modified approach. I now have a cron event, that triggers a lambda each minute. The lambda then gets the message from the queue. The modification is, that the cron job gets disabled when the queue is empty. If elements are added it is enabled again etc. Thank you all!

已提问 4 年前2582 查看次数
1 回答
0
已接受的回答

You could also have a lambda that is triggered by a timer (cloudwatch event) every 30s. It would read one item from the queue or exit immediately if there are no messages on the queue. One invocation every 30s is fairly cheap, and you are guaranteed to have no more than that rate of scrapes.

profile pictureAWS
已回答 4 年前
profile picture
专家
已审核 9 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则