EC2 sqs consumer different timeouts

0

Hi,

I'm using the SQS-Consumer for getting messages from an AWS SQS (on an EC2 instance I'm running a simple app using pm2). Once the message is received, I'm grabbing relevant data and call an external function called messageHandler which does some operations on the message (scrape the URL from message using Puppeteer) and then update DB accordingly:

const app = Consumer.create({
 ...

 handleMessage: async (message) => {
 const { id, protocol, hostname, pathname } = JSON.parse(message.Body) as QueueMessageBody;
 const res = await puppeteerSinglePageCrawl(id, url.protocol, url.hostname, url.pathname, logStreamName);
 return Promise.resolve();
}

My problem is that when a message is read from the queue, sometimes I get Timeout Errors on opening the page with Puppeteer:

await this.page.goto(`${protocol}//${hostname}${pathname}`, {
 waitUntil: 'networkidle2',
 referer: 'https://www.google.com/',
 timeout: 20000,
});

However, when I connect to my EC2 instance via SSH and call the same function manually ts-node messageHandler.ts I'm not getting the timeout error.

My initial thought was that the issue might be with the waitUntil but clearly when called manually I don't get this error and the page is opening correctly.

My second thought was that somehow the network might be overloaded on the EC2 instance when the consumer is running, but I've been testing the manual calls while the consumer was running separately and I still got different (successful) results on manual execution.

What might be the reason for this?

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions