List queue messages yet unprocessed by consumer

0

Hi everyone,

I have a first lambda than scans an ftp server for new files every 2 minutes. Once a new file is detected, the lambda function will send its location path as a message in an SQS queue.

A second lambda will get triggered as soon as a message is in the queue. It will receive this message containing the location of the file and download it from the server to its tmp folder, then transfer it from its tmp folder to an S3 bucket.

To avoid the first lambda to pass the same files locations multiple times, I make it first check the file is not already in the bucket and not in the queue neither. The problem is that the using: sqs.receive_message() doesn't return anything in my first lambda. I assume it is due to the visibility timeout.

So my question is: Is there a way i can peek in the queue for messages that have not yet been processed by the consumer (second lambda)?

Would setting the visibility timeout to 0 (if possible) do the trick? If I then call sqs.receive_message() in my first lambda (not the consumer then), will that remove the messages from the queue?

Many thanks

Antoine

  • Have you looked into AWS Step Functions yet? I think it might solve a lot of your problems regarding chaining Lambda functions together.

已提问 2 年前702 查看次数
1 回答
1

SQS does not have a way to peek messages in the queue. You should not follow this approach. If you want to make sure that each file is processed only once and that even the first function doesn't put the same file name twice into the queue, consider using SQS FIFO queues with deduplication.

profile pictureAWS
专家
Uri
已回答 2 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则