NLB Distribution unevenly with wildfly and pgBouncer

Question

Hello community,

I cannot seem to find the answer for my current problem.
The setup is that I have a number of wildfly server running connecting to an NLB where 2 instances of pgBouncer are running.
We recently had a traffic spike and I noticed that one of the 2 pgBouncer instances had a lot more "packages out" than the other (10x more). From that point it seems that the second instance was almost not hit at all.
The postgres in the background seems to confirm this as the maximum number of connections was as high as configured by a single pgBouncer instance and the second did not seem to establish any connection. (or only very few)

The pgBouncer are working fine, my suspicion is the NLB. I read about long lasting connections leading to problems in the load distribution.

Is the NLB the correct Load balancer for this scenario or should we move to the ALB? Is there anything I could configure/test for this scenario? I do not know where I even should start investing the issue further.

Thanks!

Answer

NLB distributes connections using flow hash algorithm based on the protocol, source IP address, source port, destination IP address, destination port, and TCP sequence number. That means for example if you have a single long lived TCP connection, it will always be routed to same target. The balance becomes evident as you open more connections, that should be routed to the other targets.

I am mentioning this because DB connections from Application to Database tend to be long lived, and if you don't have many clients (as in, the application that connects to the NLB) you might experience an non optimal balance.

A few things to check:
- If multi-az, are your clients connecting to both NLB IP's? You can either check this on the clients or using the VPC flow logs.
- Check in your client connecting to NLB, if there are DNS options you can modify (such as avoid DNS caching)
- Have you enabled Cross-zone balancing in your NLB? This can help in case your clients are favoring one NLB node (IP).

Lastly, ALB only supports HTTP/HTTPS, so it will not work for PostgreSQL database connections.

NLB Distribution unevenly with wildfly and pgBouncer

Contenido relevante