Intermittent InvocationTargetException connecting to S3

Question

Hello,  
  
I have a Lambda that processes S3 events (triggered by the creation of files in a specified directory in the S3 bucket) from an SQS queue.  
  
The Lambda's processing of single S3 event (i.e. creating one file) works as expected.  
  
If I create a batch of 10 files at the same time, multiple instances of the Lambda are initiated. Some will work without issue but at least one of these (and some times several) will fail. The behaviour is inconsistent.  
  
During processing, the first error occurs when the Lambda tries to connect to the Secrets Manager:

```
com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - connecting to secretsmanager.ap-southeast-2.amazonaws.com/:
c.a.http.conn.ClientConnectionManagerFactory - java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... stack trace...
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to secretsmanager.ap-southeast-2.amazonaws.com: [secretsmanager.ap-southeast-2.amazonaws.com/, secretsmanager.ap-southeast-2.amazonaws.com/, secretsmanager.ap-southeast-2.amazonaws.com/] failed: connect timed out
... stack trace...
Caused by: java.net.SocketTimeoutException: connect timed out
```

The connection is retried a couple of further times but always fails. The Lambda code catches the exception and tries to do some cleaning up but cannot connect to S3:

```
com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - Connecting socket to .s3.ap-southeast-2.amazonaws.com/: with timeout 10000
c.a.http.conn.ClientConnectionManagerFactory - java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... stack trace...
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to .s3.ap-southeast-2.amazonaws.com: [.s3.ap-southeast-2.amazonaws.com/] failed: connect timed out
... stack trace...
Caused by: java.net.SocketTimeoutException: connect timed out
```

I am using the following libraries:  
aws-lambda-java-core: 1.2.0  
aws-java-sdk-s3: 1.11.714  
aws-java-sdk-events: 1.11.714  
aws-java-sdk-secretsmanager: 1.11.718  
aws-java-sdk-sqs: 1.11.719  
  
Thanks in advance for any assistance.  
  
Edited by: MarkHarrisIntellihub on Feb 17, 2020 11:56 AM

Answer

The issue was a networking one - one of the private subnets that the Lambda's VPC uses had a mis-configured route table that was assigned to a non-existent NAT gateway.  
  
Once the correct NAT gateway was added, the Lambda worked as expected.

Intermittent InvocationTargetException connecting to S3

Contenuto pertinente