EC2 sqs consumer different timeouts

0

Hi,

I'm using the SQS-Consumer for getting messages from an AWS SQS (on an EC2 instance I'm running a simple app using pm2). Once the message is received, I'm grabbing relevant data and call an external function called messageHandler which does some operations on the message (scrape the URL from message using Puppeteer) and then update DB accordingly:

const app = Consumer.create({
 ...

 handleMessage: async (message) => {
 const { id, protocol, hostname, pathname } = JSON.parse(message.Body) as QueueMessageBody;
 const res = await puppeteerSinglePageCrawl(id, url.protocol, url.hostname, url.pathname, logStreamName);
 return Promise.resolve();
}

My problem is that when a message is read from the queue, sometimes I get Timeout Errors on opening the page with Puppeteer:

await this.page.goto(`${protocol}//${hostname}${pathname}`, {
 waitUntil: 'networkidle2',
 referer: 'https://www.google.com/',
 timeout: 20000,
});

However, when I connect to my EC2 instance via SSH and call the same function manually ts-node messageHandler.ts I'm not getting the timeout error.

My initial thought was that the issue might be with the waitUntil but clearly when called manually I don't get this error and the page is opening correctly.

My second thought was that somehow the network might be overloaded on the EC2 instance when the consumer is running, but I've been testing the manual calls while the consumer was running separately and I still got different (successful) results on manual execution.

What might be the reason for this?

已提問 1 年前檢視次數 50 次
沒有答案

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南