I have recently started seeing many errors while running Lambda functions. General process we follow using node.js scripts:
- Send event file with two attachments, one file for delivery plus one json file for instructions
- Upload file to S3 bucket,
- Send the file to an ftp server or in an email as an attachment.
The errors started occurring within the last week or so, everything was running fine prior to that for a number of months. To the best of my knowledge nothing in our configuration has changed except adding more services to our account (some ec2 instances). In some cases it looks like the file transfer worked fine, but then I get the error message after processing, with the previous requestid referenced. (see below chain of messages). In other cases the error occurs and then the file is processed fine a few seconds later.
Example Flow with error:
START RequestId: e282d989-e488-11e6-9e54-0d7f64db0dd8 Version: $LATEST
2017-01-27T12:05:24.984Z e282d989-e488-11e6-9e54-0d7f64db0dd8 Received event:
2017-01-27T12:05:24.985Z e282d989-e488-11e6-9e54-0d7f64db0dd8
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "2017-01-27T12:05:24.791Z",
"eventName": "ObjectCreated:Put", *--- other file details etc*
2017-01-27T12:05:24.985Z e282d989-e488-11e6-9e54-0d7f64db0dd8 DownloadFile (bucket name here) (file name here)
2017-01-27T12:05:25.077Z e282d989-e488-11e6-9e54-0d7f64db0dd8 Found config email bucket JSON file [(file name here)]
2017-01-27T12:05:25.077Z e282d989-e488-11e6-9e54-0d7f64db0dd8 Data read from file =
2017-01-27T12:05:25.077Z e282d989-e488-11e6-9e54-0d7f64db0dd8 email bucket info = (bucket name here)
2017-01-27T12:05:25.159Z e282d989-e488-11e6-9e54-0d7f64db0dd8 Uploaded file [(file name here)] to bucket (bucket name here)
2017-01-27T12:05:26.434Z e282d989-e488-11e6-9e54-0d7f64db0dd8 Successfully ftp file to (ip addr here)
END RequestId: e282d989-e488-11e6-9e54-0d7f64db0dd8
REPORT RequestId: e282d989-e488-11e6-9e54-0d7f64db0dd8 Duration: 1455.55 ms Billed Duration: 1500 ms Memory Size: 1024 MB Max Memory Used: 40 MB
START RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Version: $LATEST *--- start of new request*
2017-01-27T12:05:49.065Z e282d989-e488-11e6-9e54-0d7f64db0dd8 Error: read ECONNRESET
at errnoException (net.js:905:11)
at TCP.onread (net.js:559:19) *----- failure referencing previous request id*
END RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8
REPORT RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Duration: 38.32 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 40 MB
RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Process exited before completing request
START RequestId: f0e89c6d-e488-11e6-a6b6-cd817fd75da8 Version: $LATEST *--- retries previous request to email next file and everything continues on as normal*
Is there a timeout setting I need to increase, or are we not closing connections properly after ftp transfer?? Any help would be appreciated. I don't believe we have a functional problem because the file eventually gets through, but the errors cost time to verify and are causing clutter in my monitoring.
Thanks in advance.