How to make ALB seamlessly re-forward to a healthy target?

0

How to make ALB seamlessly and instantly re-forward HTTP request to a healthy target?

I am trying to configure ALB to seamlessly forward HTTP requests to another healthy target when the original target is unavailable, but not yet marked unhealthy. But I get Gateway error 502 until 2 unsuccessful healthchecks happen and the target is marked unhealthy. I have 1 ALB, 1 target group, 2 targets that pointed to 2 IPs in separate VM instances. Session stickyness is on. App cookie JSESSIONID. Tomcat is configured for Clustering. Scenario:

  1. start Tomcat at node A
  2. browse app http: //My-ALB-link/examples/jsp/jsp2/simpletag/hello.jsp. Node A responds.
  3. start Tomcat at node B. wait until ALB marks target healthy
  4. stop Tomcat at node A and quickly refresh browser - get error 502 Bad Gateway
  5. wait a minute, refresh browser. Node B responds.

How to configure ALB to instantly re-forward HTTP request to another node if the first node happen to be unavailable? User's wont wait while ALB makes 2 healthchecks and marks a node unhealthy.

thanks, Mark

asked 2 years ago1258 views
4 Answers
0

I think you already have your own answer here. Because ALB thinks the target is healthy it will continue to forward requests to it. ALB won't "reforward" a request because it doesn't know that there is a problem yet. Even then, ALB doesn't send requests twice so in the event that a back-end has failed the client will need to resend the request.

You can modify the health check timers but it sounds like you want a way to tell ALB that a target is unhealthy straight away.

Something you might try is manually/automatically deregistering the target: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/elbv2/deregister-targets.html

profile pictureAWS
EXPERT
answered 2 years ago
  • ALB thinks the target is healthy it will continue to forward requests to it.

    That is the problem. ALB receives "bad gateway" from network layer and continues to think that the target is healthy. It looks like a load balancing subprogram is decoupled from a health check subprogram. Where only the latter can set a target status, and the former can only read it, and does not paying attention an actual status of the target.

    ALB won't "reforward" a request because it doesn't know that there is a problem yet.

    It knows, it cannot connect to IP/port, it received error from network layer.

    Even then, ALB doesn't send requests twice so in the event that a back-end has failed the client will need to resend the request.

    I thought in case of ALB, it works like sort of proxy, receives full HTTP request from Client, then sends it to a target. If that failed, ALB can theoretically send it again to another target. Client is still awaiting a response. I may be wrong, need to read more how ALB works.

    Yes, I've already modified timers to minimum period. Again, there is no need to additionally tell ALB that a target is not healthy. It already got it. The monitor works same way, it connects and reads from target, if target does not response, the monitor counts this as a failure and sets the status to unhealthy. What is a difference with the LB calls? It calls target similar way and receives same error from network. From that it already knows the actual status of a target.

  • continue ...

    Something you might try is manually/automatically deregistering the target:

    Yes, I also thought about automatically deregistering. But that would require to make my own monitor, which also will be timer based, most likely. Because there are no events/triggers from ALB I could set my handler on. And "my monitor/deregistrator" wont make ALB to re-forward request.

  • I thought in case of ALB, it works like sort of proxy

    While ALB could do that, it doesn't operate that way. Sending requests to a targets multiple times could potentially set up patterns where you need to handle the case of a request landing at the back end multiple times and ensure idempotency. Rather than do that, ALB only sends requests once. It's up to the client software to determine whether it should retry or not based on the return code.

0

@Brettski

While ALB could do that, it doesn't operate that way. Sending requests to a targets multiple times could potentially set up patterns where you need to handle the case of a request landing at the back end multiple times and ensure idempotency. Rather than do that, ALB only sends requests once. It's up to the client software to determine whether it should retry or not based on the return code.

I agree if the request was sent, and no response from the target, then it can be wrong to re-send. But in the scenario I am talking about ALB cannot even send the request at first time. It cannot establish TCP connection to send it. In such case, from my perspective, it will be safe to send it to another target.

answered 2 years ago
  • But: ALB doesn't know the request has failed. It thinks the target is healthy therefore it sends the request. Only then it gets back an error code; it's too late that point. If the host refused the TCP connection that would be a different matter. But that's not the scenario here.

0

Please tell me how ALB can send HTTP request without first establishing TCP connection with a target? It sends SYN, should receive SYN-ACK from the target. It will not send further packets (with HTTP payload) if it does not receive SYN-ACK and good handshake is not established.

It is exactly the scenario described in the original message. See item 4.

answered 2 years ago
  • In this case, ALB is creating the 502 Bad Gateway response because the target failed to respond. It was still marked healthy so ALB sent a request to it. To go back to the original answer: ALB does not resend requests. That's not something that is changeable here. If you'd like that behaviour changed (and there are no guarantees here); reach out to your local AWS Solutions Architect and have them raise a feature request with the ALB team.

0

Yes, it is probably good idea to raise a feature request. With Support first.

By the way, Network LB has the feature that looks like what I need - passive health checks. I have not tried that yet. If it works, may be I will need to chain them something like that: ALB -> NLB => Targets.

It is different question - do you know what redundancy an ALB listener has behind the scene? Can it fail? What happens when it fails?

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions