Long response time for cloudfront misses

0

Need some help debugging this long response time I'm seeing from my cloudfront CDN for images that have not been cached. The outline of our setup is that we have a cloudfront cdn that responds with cached images when available. If no cached image is available, there's a lambda that pulls the requested image from s3 and resizes it using sharp.js, then sends the resized image as the response to the request. Cloudfront caches this image and then uses it for subsequent requests for the same image. The problem is that this handling takes 2-3s usually. You can see in this screenshot.

I'm only partially aware of the breakdown of those 2-3s. That screenshot is of logs from cloudfront, so that means the problem must lie somewhere within our cloudfront setup. The lambda itself takes 800-1300ms from start to finish, and that includes the time it takes to pull the image from s3, resize it, convert it to a buffer, then respond to the request. We already use the http keepAlive optimization to reduce the latency of pulling the image from s3. However the lambda's running time is often 50% or less of the total response time, so that means there's another significant bottleneck elsewhere that I haven't discovered, and I'm not sure how to go about finding it. I've tried enabling AWS X-Ray to get more insights into the problem but our lambda is on Lambda@Edge, which doesn't support X-Ray. What else can I investigate and where else could I look?

  • Hi! Have you solved the issue ? I am experiencing the same problem...Thanks for your help.

4 Answers
0

Hi there,

Would it be possible for you to copy the code you have on the Lambda@Edge to a normal Lambda? If you do that, then you could use our AWS Distro for OpenTelemetry (ADOT) Lambda Layers to profile your code and send traces to X-Ray to see where your bottlenecks are occurring for the Lambda part of your code.

Although Lambda may not be the exact same as Lambda@Edge, it might help you narrow down the cause for the slowdown. With the Lambda Layers we distribute for JavaScript, all you have to do is add your Lambda Layer and invoke your Lambda function. As long as it has permissions to send traces to X-Ray (the easiest way is to enable active tracing) then you'll see traces for your downstream S3 calls in the X-Ray console. The documentation I liked has options to trace the call in non-Lambda environments as well.

Hopefully you can try that and let us know if it helps you identify any slowdowns!

AWS
answered 2 years ago
0

Thanks for the response! Is there a doc/guide that explains how to move from Lambda@Edge to normal lambda for a cloudfront distribution? The function associations are defined in the behaviors of the distribution, and for origin response it only gives me two options: "No Associations" and "Lambda@Edge". It seems I would need a third that would be something like just "Lambda", is that right? I'm still learning my way around aws so I'd greatly appreciate some direction, thank you!

answered 2 years ago
0

Even though Lambda@Edge does not support X-Ray, does it by any chance support Lambda Layers? I'm assuming it does not, and I guess I mentioned this above already, but if it did you could use our documentation for adding OpenTelemetry Lambda Layers independent of X-Ray.

I'm not aware of any documentation for moving from Lambda@Edge to Lambda :/ But is your Lambda@Edge function really complicated? If it were simple enough then you could just copy and paste the function code a fresh new Lambda? It's not ideal but that should let you profile your code at least...

Towards OpenTelemetry, you might be interested in this Step by Step guide on adding OpenTelemetry to a Python Lambda I wrote just a few months ago. Even if you're not using Python the steps should be very similar!

Glad you're exploring AWS! :) I'll mention to my team that you'd like a tracing solution for Lambda@Edge, and we'll look for the opportunity to provide this enhancement!

AWS
answered 2 years ago
0

In case you are still experiencing this problem, the recent release of the server-timing response header may be helpful in trying to track down where the latency is occurring. You can find further info on the server-timing header here: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-response-headers-policies.html#server-timing-header

This is also something that AWS support can help you with, if you are still experiencing problems I would recommend opening a support case.

AWS
EXPERT
Paul_L
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions