List queue messages yet unprocessed by consumer

0

Hi everyone,

I have a first lambda than scans an ftp server for new files every 2 minutes. Once a new file is detected, the lambda function will send its location path as a message in an SQS queue.

A second lambda will get triggered as soon as a message is in the queue. It will receive this message containing the location of the file and download it from the server to its tmp folder, then transfer it from its tmp folder to an S3 bucket.

To avoid the first lambda to pass the same files locations multiple times, I make it first check the file is not already in the bucket and not in the queue neither. The problem is that the using: sqs.receive_message() doesn't return anything in my first lambda. I assume it is due to the visibility timeout.

So my question is: Is there a way i can peek in the queue for messages that have not yet been processed by the consumer (second lambda)?

Would setting the visibility timeout to 0 (if possible) do the trick? If I then call sqs.receive_message() in my first lambda (not the consumer then), will that remove the messages from the queue?

Many thanks

Antoine

  • Have you looked into AWS Step Functions yet? I think it might solve a lot of your problems regarding chaining Lambda functions together.

1개 답변
1

SQS does not have a way to peek messages in the queue. You should not follow this approach. If you want to make sure that each file is processed only once and that even the first function doesn't put the same file name twice into the queue, consider using SQS FIFO queues with deduplication.

profile pictureAWS
전문가
Uri
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠