High Latency OpenSearch Behind ALB

0

I am trying to set up OpenSearch in a private VPC subnet behind a Load Balancer in a public subnet. The load balancer endpoint is in turn placed in a Cloudfront distribution. Right now I am testing this with HTTP -- will try HTTPS once we are able to set up our DNS. After configuring the security groups to allow OpenSearch and the ALB to communicate, and after adding the listener/target group, I am able to connect to OpenSearch through the load balancer endpoint. However, if I try to access via the Cloudfront endpoint, I get a 504 Error: The Request Could Not Be Satisfied. I try pinging the ALB endpoint via curl and notice that it is taking 75 seconds to respond with 200-OK. So it seems that Cloudfront is not responding due to late responses from the Load Balancer. It always takes exactly 75 seconds -- except sometimes when I fire up the cluster, the first response comes back in a fraction of a second as it should, then on all subsequent attempts it takes 75 seconds. I am in Maryland and the cluster is set up in the Oregon region. I tried this with three progressively larger instances of OpenSearch and the compute power made no difference. I've been trying to figure this out for weeks -- any suggestions on what I am doing wrong? Thanks!

  • I chatted with support and we found the issue. I had accidentally configured one of the ALB subnets to private when they both needed to be public. Problem solved!

2 Answers
0

Just a thought, but is it possible to increase the origin timeout in CloudFront. When you configure the ALB endpoint as an origin, check the advanced settings and notice that there is a time. By default I believe it is 30 seconds, but it can be increased to 60, and beyond with a quota increase request.

https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-custom-origin-response/ https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesOriginResponseTimeout

profile pictureAWS
answered a year ago
  • Thanks! I have tried that up to the 60 seconds without a quota increase, but it didn't work. I checked the link for getting the quota increase and cannot see Cloudfront in the list of available services. Besides, I would need a timeout over 75 seconds since that is how long the response from ALB takes. And ultimately I need to understand why it would take 75 seconds -- the goal is sub-second latency which should be achievable with this setup, especially when its not under any kind of load.

0

I suggest to enable Server-Timing header and check cdn-upstream-connect and cdn-upstream-fbl metrics to see which one causing the added latency. In addition, ALB access logs can help to verify whether the request is delayed by ALB processing or by response from the target

AWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions