By using AWS re:Post, you agree to the Terms of Use
/How to make ALB seamlessly re-forward to a healthy target?/

How to make ALB seamlessly re-forward to a healthy target?

0

How to make ALB seamlessly and instantly re-forward HTTP request to a healthy target?

I am trying to configure ALB to seamlessly forward HTTP requests to another healthy target when the original target is unavailable, but not yet marked unhealthy. But I get Gateway error 502 until 2 unsuccessful healthchecks happen and the target is marked unhealthy. I have 1 ALB, 1 target group, 2 targets that pointed to 2 IPs in separate VM instances. Session stickyness is on. App cookie JSESSIONID. Tomcat is configured for Clustering. Scenario:

  1. start Tomcat at node A
  2. browse app http: //My-ALB-link/examples/jsp/jsp2/simpletag/hello.jsp. Node A responds.
  3. start Tomcat at node B. wait until ALB marks target healthy
  4. stop Tomcat at node A and quickly refresh browser - get error 502 Bad Gateway
  5. wait a minute, refresh browser. Node B responds.

How to configure ALB to instantly re-forward HTTP request to another node if the first node happen to be unavailable? User's wont wait while ALB makes 2 healthchecks and marks a node unhealthy.

thanks, Mark

3 Answers
0

I think you already have your own answer here. Because ALB thinks the target is healthy it will continue to forward requests to it. ALB won't "reforward" a request because it doesn't know that there is a problem yet. Even then, ALB doesn't send requests twice so in the event that a back-end has failed the client will need to resend the request.

You can modify the health check timers but it sounds like you want a way to tell ALB that a target is unhealthy straight away.

Something you might try is manually/automatically deregistering the target: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/elbv2/deregister-targets.html

EXPERT
answered 7 days ago
  • ALB thinks the target is healthy it will continue to forward requests to it.

    That is the problem. ALB receives "bad gateway" from network layer and continues to think that the target is healthy. It looks like a load balancing subprogram is decoupled from a health check subprogram. Where only the latter can set a target status, and the former can only read it, and does not paying attention an actual status of the target.

    ALB won't "reforward" a request because it doesn't know that there is a problem yet.

    It knows, it cannot connect to IP/port, it received error from network layer.

    Even then, ALB doesn't send requests twice so in the event that a back-end has failed the client will need to resend the request.

    I thought in case of ALB, it works like sort of proxy, receives full HTTP request from Client, then sends it to a target. If that failed, ALB can theoretically send it again to another target. Client is still awaiting a response. I may be wrong, need to read more how ALB works.

    Yes, I've already modified timers to minimum period. Again, there is no need to additionally tell ALB that a target is not healthy. It already got it. The monitor works same way, it connects and reads from target, if target does not response, the monitor counts this as a failure and sets the status to unhealthy. What is a difference with the LB calls? It calls target similar way and receives same error from network. From that it already knows the actual status of a target.

  • continue ...

    Something you might try is manually/automatically deregistering the target:

    Yes, I also thought about automatically deregistering. But that would require to make my own monitor, which also will be timer based, most likely. Because there are no events/triggers from ALB I could set my handler on. And "my monitor/deregistrator" wont make ALB to re-forward request.

  • I thought in case of ALB, it works like sort of proxy

    While ALB could do that, it doesn't operate that way. Sending requests to a targets multiple times could potentially set up patterns where you need to handle the case of a request landing at the back end multiple times and ensure idempotency. Rather than do that, ALB only sends requests once. It's up to the client software to determine whether it should retry or not based on the return code.

0

Per the documentation below are some of the possible reasons for 502 bad gateway error

Troubleshoot your Application Load Balancers

HTTP 502: Bad gateway

Possible causes:

  • The load balancer received a TCP RST from the target when attempting to establish a connection.

  • The load balancer received an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)", when attempting to establish a connection. Check whether traffic is allowed from the load balancer subnets to the targets on the target port.

  • The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.

  • The target response is malformed or contains HTTP headers that are not valid.

  • The load balancer encountered an SSL handshake error or SSL handshake timeout (10 seconds) when connecting to a target.

  • The deregistration delay period elapsed for a request being handled by a target that was deregistered. Increase the delay period so that lengthy operations can complete.

answered 7 days ago
  • Tushar, in my case 502 is because I've stopped the target that ALB is stuck to and its health check not yet happen to set it unhealthy.

    Somehow in similar situation Apache HTTP server can re-forward HTTP request to another target, instantly and seamlessly. You can see what do I mean in this video https://youtu.be/9gtpyqhd-NI?t=859

    I expected ALB can do something similar.

0

@Brettski

While ALB could do that, it doesn't operate that way. Sending requests to a targets multiple times could potentially set up patterns where you need to handle the case of a request landing at the back end multiple times and ensure idempotency. Rather than do that, ALB only sends requests once. It's up to the client software to determine whether it should retry or not based on the return code.

I agree if the request was sent, and no response from the target, then it can be wrong to re-send. But in the scenario I am talking about ALB cannot even send the request at first time. It cannot establish TCP connection to send it. In such case, from my perspective, it will be safe to send it to another target.

answered 2 days ago
  • But: ALB doesn't know the request has failed. It thinks the target is healthy therefore it sends the request. Only then it gets back an error code; it's too late that point. If the host refused the TCP connection that would be a different matter. But that's not the scenario here.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions