400 ms difference between APIG latency and Lambda Latency

Question

We have a Lambda with multiple functions fronted by an APIG. We expect that the APIG latency  for a specific lambda endpoint (resource) should be very close to the p50 latency reported by the lambda for that endpoint.

However we consistently see a ~400 ms difference between the APIG latency and the various latencies for each lambda function behind the APIG.

Two questions:

1. What could be causing this and how can I test for this?

2. One hypothesis was that it might be due to SSL. I expect that SSL would only add 100 ms not 400 ms. That being said how can I confirm if the is SSL between the APIG and the Lambda, and if there is how much latency that is adding?

Accepted Answer

There are a number of possible factors so I won't try to guess on what the cause would be. I'd recommend utilizing X-Ray tracing on API Gateway and your lambdas. This will give you insight into where the time is being spent and some hints as to what might be going on.

Some resources to get started on getting AWS X-Ray integrated:

Amazon API Gateway active tracing support

AWS Lambda and AWS X-Ray

Some resources to get started on getting AWS X-Ray integrated:

[Amazon API Gateway active tracing support](https://docs.aws.amazon.com/xray/latest/devguide/xray-services-apigateway.html)

[AWS Lambda and AWS X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html)

400 ms difference between APIG latency and Lambda Latency

Relevant content