Lamdba ECONNRESET errors occurring regularly

0

I have recently started seeing many errors while running Lambda functions. General process we follow using node.js scripts:

  1. Send event file with two attachments, one file for delivery plus one json file for instructions
  2. Upload file to S3 bucket,
  3. Send the file to an ftp server or in an email as an attachment.

The errors started occurring within the last week or so, everything was running fine prior to that for a number of months. To the best of my knowledge nothing in our configuration has changed except adding more services to our account (some ec2 instances). In some cases it looks like the file transfer worked fine, but then I get the error message after processing, with the previous requestid referenced. (see below chain of messages). In other cases the error occurs and then the file is processed fine a few seconds later.

Example Flow with error:

START RequestId: e282d989-e488-11e6-9e54-0d7f64db0dd8 Version: $LATEST
2017-01-27T12:05:24.984Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	Received event:
2017-01-27T12:05:24.985Z	e282d989-e488-11e6-9e54-0d7f64db0dd8
{
    "Records": [
        {
            "eventVersion": "2.0",
            "eventSource": "aws:s3",
            "awsRegion": "us-east-1",
            "eventTime": "2017-01-27T12:05:24.791Z",
            "eventName": "ObjectCreated:Put",  *--- other file details etc*

2017-01-27T12:05:24.985Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	DownloadFile (bucket name here) (file name here)

2017-01-27T12:05:25.077Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	Found config email bucket JSON file [(file name here)]

2017-01-27T12:05:25.077Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	Data read from file =

2017-01-27T12:05:25.077Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	email bucket info = (bucket name here) 

2017-01-27T12:05:25.159Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	Uploaded file [(file name here)] to bucket (bucket name here) 

2017-01-27T12:05:26.434Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	Successfully ftp file to (ip addr here)

END RequestId: e282d989-e488-11e6-9e54-0d7f64db0dd8

REPORT RequestId: e282d989-e488-11e6-9e54-0d7f64db0dd8	Duration: 1455.55 ms	Billed Duration: 1500 ms Memory Size: 1024 MB	Max Memory Used: 40 MB	

START RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Version: $LATEST  *--- start of new request*

2017-01-27T12:05:49.065Z	e282d989-e488-11e6-9e54-0d7f64db0dd8	Error: read ECONNRESET
at errnoException (net.js:905:11)
at TCP.onread (net.js:559:19)    *----- failure referencing previous request id*

END RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8

REPORT RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8	Duration: 38.32 ms	Billed Duration: 100 ms Memory Size: 1024 MB	Max Memory Used: 40 MB	

RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Process exited before completing request

START RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Version: $LATEST   *--- retries previous request to email next file and everything continues on as normal*

Is there a timeout setting I need to increase, or are we not closing connections properly after ftp transfer?? Any help would be appreciated. I don't believe we have a functional problem because the file eventually gets through, but the errors cost time to verify and are causing clutter in my monitoring.

Thanks in advance.

asked 8 years ago4K views
1 Answer
1

Note that port25 egress is throttled on all AWS compute services; if you are sending significant volumes of email, we recommend using Amazon SES. AWS Lambda does not currently support raising port25 egress limits on a per-function or per-account basis.

Also, if you're using NodeJS, make sure that you have completed all asynchronous processing prior to returning from your request handler; otherwise, you may have a socket connection that is frozen (and subsequently thawed) across requests, which will often lead to a connection reset. Async processing in NodeJS is a frequent source of this type of error.

Hope this helps!

answered 7 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions