Intermittent InvocationTargetException connecting to S3

Hello,

I have a Lambda that processes S3 events (triggered by the creation of files in a specified directory in the S3 bucket) from an SQS queue.

The Lambda's processing of single S3 event (i.e. creating one file) works as expected.

If I create a batch of 10 files at the same time, multiple instances of the Lambda are initiated. Some will work without issue but at least one of these (and some times several) will fail. The behaviour is inconsistent.

During processing, the first error occurs when the Lambda tries to connect to the Secrets Manager:

com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - connecting to secretsmanager.ap-southeast-2.amazonaws.com/<ip>:<port>
c.a.http.conn.ClientConnectionManagerFactory - java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... stack trace...
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to secretsmanager.ap-southeast-2.amazonaws.com:<port> [secretsmanager.ap-southeast-2.amazonaws.com/<ip>, secretsmanager.ap-southeast-2.amazonaws.com/<ip>, secretsmanager.ap-southeast-2.amazonaws.com/<ip>] failed: connect timed out
... stack trace...
Caused by: java.net.SocketTimeoutException: connect timed out

The connection is retried a couple of further times but always fails. The Lambda code catches the exception and tries to do some cleaning up but cannot connect to S3:

com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - Connecting socket to <s3 bucket>.s3.ap-southeast-2.amazonaws.com/<ip>:<port> with timeout 10000
c.a.http.conn.ClientConnectionManagerFactory - java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... stack trace...
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to <s3 bucket>.s3.ap-southeast-2.amazonaws.com:<port> [<s3 bucket>.s3.ap-southeast-2.amazonaws.com/<ip>] failed: connect timed out
... stack trace...
Caused by: java.net.SocketTimeoutException: connect timed out

I am using the following libraries:
aws-lambda-java-core: 1.2.0
aws-java-sdk-s3: 1.11.714
aws-java-sdk-events: 1.11.714
aws-java-sdk-secretsmanager: 1.11.718
aws-java-sdk-sqs: 1.11.719

Thanks in advance for any assistance.

Edited by: MarkHarrisIntellihub on Feb 17, 2020 11:56 AM