NLB not distributing traffic between 2 servers

See point 2.

Cross-Zone Load Balancing: By default, NLB does not enable Cross-Zone Load Balancing (CZLB). If your mail servers are in different Availability Zones, this could lead to uneven distribution. Consider enabling CZLB, but be aware that this may incur additional data transfer charges.

The issue you're experiencing with your Network Load Balancer (NLB) not distributing traffic evenly between your two mail servers could be due to several factors. Let's address the potential causes and solutions:

1. Load Balancing Algorithm: NLB uses a flow hash algorithm to route traffic. This means that for a given set of parameters (source IP, source port, destination IP, destination port), traffic will consistently route to the same target. This can sometimes lead to an uneven distribution, especially if you have a small number of clients or if most of your traffic comes from a limited set of source IPs.

2. Cross-Zone Load Balancing: By default, NLB does not enable Cross-Zone Load Balancing (CZLB). If your mail servers are in different Availability Zones, this could lead to uneven distribution. Consider enabling CZLB, but be aware that this may incur additional data transfer charges.

3. Health Checks: The behavior you're seeing where requests trickle to mail server 2 when server 1 is marked unhealthy is expected. NLB has a fail-open mode where it routes requests to all registered targets (including unhealthy ones) when all targets are considered unhealthy.

4. Connection Draining: When replacing mail server 1, you should implement a strategy to gradually drain connections. NLB doesn't have built-in connection draining, so you'll need to manage this at the application level.

5. Intermittent Hangs: The hanging issues on port 587 could be related to the NLB's TCP idle timeout. By default, this is set to 350 seconds. You might want to adjust this setting to better suit your application's needs.

To address these issues and prepare for replacing mail server 1:

1. Check your target group settings and ensure both servers are properly registered and passing health checks.

2. Consider enabling Cross-Zone Load Balancing if your servers are in different Availability Zones.

3. Implement a strategy to gradually drain connections from mail server 1 before taking it offline.

4. Monitor NLB metrics like ActiveFlowCount, HealthyHostCount, and UnHealthyHostCount to ensure proper distribution and health of your targets.

5. Adjust the TCP idle timeout if necessary to address the hanging issues.

6. If possible, implement a canary or synthetic monitoring to test the behavior when transitioning between servers.

By taking these steps, you can improve the traffic distribution and minimize the risk when replacing mail server 1. Remember to thoroughly test your setup before making any production changes.
Sources
Network Load Balancer (NLB) troubleshooting guide | AWS re:Post
Troubleshoot your Network Load Balancer - Elastic Load Balancing
Introducing NLB TCP configurable idle timeout | Networking & Content Delivery
Elastic Load Balancer: Maximizing Benefits and Keeping Costs Low | Networking & Content Delivery
Application, Network, and Gateway Load Balancing - Difference Between Load Balancing Types - AWS