How to reduce the time it takes a request to pass from a ALB to the actual Fargate Server?

0

Hey, I have a webhook endpoint where our service provider send a payload which I have to respond to within 2 seconds. I've been getting way too many timeout errors from the service provider, meaning I wasn't able to respond within 2 seconds.

I did some digging as to when the Fargate Server gets the payload vs when the ALB receives it. I went through some of the access logs from the ALB and found that it takes about a second or so to pass the payload from ALB to the fargate server.

Here's the timestamp at which the request arrived to the ALB - 15:19:20.01 and my server recieved it at - 15:19:21.69.

There's over a second of difference, I wanna know how to reduce it. One of the solution I thought of was that instead of registering my domain + the URI to the service provider to send webhook to, I set my IP + the URI so there's no need of forwarding done by ALB.

Let me know what you guys think.

EDIT - The solution I thought of was pretty stupid because fargate provides a new IP everytime a new task is deployed (as far as I know). Also the ALB forwards the request / payload to the ECS Target Group, just throwing this fact in as well.

  • Before we try to dive too deep into the potential cause, let's make sure that you're looking at the right data. Instead of looking at timestamps (which might reflect the time the request was completed instead of when it was received), let's look instead at the latency data published in the ALB logs. The particular fields to look at in the ALB logs are the request_processing_time and target_processing_time fields as described here: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-syntax - the 6th and 7th fields.

  • The target_processing_time field is the most important field: "The total time elapsed (in seconds, with millisecond precision) from the time the load balancer sent the request to a target until the target started to send the response headers." Also, please compare against the actual response latency from your application logs if possible; timestamps aren't enough because they are often vague. My experience is that ALB request latency to a Fargate target is in the sub-millisecond range.

1 réponse
0

Hello,

As specified by Michael_F in the comments, it is essential to identify the root cause for your latency before we jump into figuring out the solution. There could be multiple reasons for the latency. You can find out the time taken during various phases of the HTTP connection using curl as shown below.

curl -L --output /dev/null --silent --show-error --write-out 'lookup:  %{time_namelookup}\nconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nredirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n' 'google.com'

Output for the above request looks something like below:

lookup:        0.002451
connect:    0.000000
pretransfer:   0.011611
redirect:      0.016241
starttransfer: 0.068402
total:         0.075255

Using this info, you can pin-point your request latency to a specific phase in the request-response process, and further investigate the root cause.

It is also helpful to enable ALB access logs as mentioned by Michael_F in the comments.

If you are unable to figure out the problem, please feel free to reach out to AWS Support to investigate the issue by following this link: https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

I hope this info is helpful to you!

profile pictureAWS
INGÉNIEUR EN ASSISTANCE TECHNIQUE
répondu il y a 2 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions