EC2 sqs consumer different timeouts

0

Hi,

I'm using the SQS-Consumer for getting messages from an AWS SQS (on an EC2 instance I'm running a simple app using pm2). Once the message is received, I'm grabbing relevant data and call an external function called messageHandler which does some operations on the message (scrape the URL from message using Puppeteer) and then update DB accordingly:

const app = Consumer.create({
 ...

 handleMessage: async (message) => {
 const { id, protocol, hostname, pathname } = JSON.parse(message.Body) as QueueMessageBody;
 const res = await puppeteerSinglePageCrawl(id, url.protocol, url.hostname, url.pathname, logStreamName);
 return Promise.resolve();
}

My problem is that when a message is read from the queue, sometimes I get Timeout Errors on opening the page with Puppeteer:

await this.page.goto(`${protocol}//${hostname}${pathname}`, {
 waitUntil: 'networkidle2',
 referer: 'https://www.google.com/',
 timeout: 20000,
});

However, when I connect to my EC2 instance via SSH and call the same function manually ts-node messageHandler.ts I'm not getting the timeout error.

My initial thought was that the issue might be with the waitUntil but clearly when called manually I don't get this error and the page is opening correctly.

My second thought was that somehow the network might be overloaded on the EC2 instance when the consumer is running, but I've been testing the manual calls while the consumer was running separately and I still got different (successful) results on manual execution.

What might be the reason for this?

질문됨 일 년 전50회 조회
답변 없음

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠