EC2 sqs consumer different timeouts

0

Hi,

I'm using the SQS-Consumer for getting messages from an AWS SQS (on an EC2 instance I'm running a simple app using pm2). Once the message is received, I'm grabbing relevant data and call an external function called messageHandler which does some operations on the message (scrape the URL from message using Puppeteer) and then update DB accordingly:

const app = Consumer.create({
 ...

 handleMessage: async (message) => {
 const { id, protocol, hostname, pathname } = JSON.parse(message.Body) as QueueMessageBody;
 const res = await puppeteerSinglePageCrawl(id, url.protocol, url.hostname, url.pathname, logStreamName);
 return Promise.resolve();
}

My problem is that when a message is read from the queue, sometimes I get Timeout Errors on opening the page with Puppeteer:

await this.page.goto(`${protocol}//${hostname}${pathname}`, {
 waitUntil: 'networkidle2',
 referer: 'https://www.google.com/',
 timeout: 20000,
});

However, when I connect to my EC2 instance via SSH and call the same function manually ts-node messageHandler.ts I'm not getting the timeout error.

My initial thought was that the issue might be with the waitUntil but clearly when called manually I don't get this error and the page is opening correctly.

My second thought was that somehow the network might be overloaded on the EC2 instance when the consumer is running, but I've been testing the manual calls while the consumer was running separately and I still got different (successful) results on manual execution.

What might be the reason for this?

Nessuna risposta

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande