sqs event triggers lambda directly, is there a way to delay that execution by 10-20 seconds?

0

I am building a serverless scraper where I have one lambda function that enqueues n urls of m websites (n>>m) one by one into a message queue. Then I have a sqs trigger that invokes another lambda which takes the url and scrapes it.

I set the batch size of the sqs trigger to 1 and the limited the lambda to 1 concurrent invocation. The problem is, that I want to have a pause of 20-40 seconds between each scraping, but currently it applies no pause and only executes the function(takes about 10 seconds).

Is there a way to make that work or do I have to rethink my current architecture?


Update: So the problem is, that adding a delay to the messages does not solve my problem. When I use the delay, x seconds are added till each message is dequeued. But as all 3 seconds a new messages are added, the dequeue operation is just 20 seconds later so the lambda is executed at 20, 23, 26, 29, 32 instead of 20, 40, 60, 80, 100 seconds after initial enqueue.


Solution: I took a modified approach. I now have a cron event, that triggers a lambda each minute. The lambda then gets the message from the queue. The modification is, that the cron job gets disabled when the queue is empty. If elements are added it is enabled again etc. Thank you all!

1개 답변
0
수락된 답변

You could also have a lambda that is triggered by a timer (cloudwatch event) every 30s. It would read one item from the queue or exit immediately if there are no messages on the queue. One invocation every 30s is fairly cheap, and you are guaranteed to have no more than that rate of scrapes.

profile pictureAWS
답변함 4년 전
profile picture
전문가
검토됨 9달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠