2 load balancers calling each other in the same VPC

Question

I have 2 load balancers: elb-server-1 and elb-server-2. In my elb-server-1, there's an API that makes a request to elb-server-2 (microservice architecture). Sometimes, it works. Sometimes, the network can't be reached from server1 to server2. For security group, I allow all traffic for both inbound and outbound. Note: when restarting the server, they can connect again, but then can't connect. I'm not sure about this behavior and how to fix it. It can't be related to quotas because I only make a few requests in a day. How to resolve this?

Accepted Answer

That's weird! After hours of debugging, I figured that using org.apache.http.client.HttpClient to call another endpoint somehow timeout after couple requests. I changed to RestTemplate and it seems to resolve the issues. I still don't know about the root cause though.

Answer

After couple of debugs, I figured that there's probably nothing wrong with the routing between 2 ELBs. The server2 has logics to send multiple messages to SNS topic. 1 Message is fine, but the 2nd message got timeout and somehow after that, the server1 can't call the server2 again and keep timeout until I restarted both server. So there's probably something wrong with sending multiple SNS messages in 1 API.

2 load balancers calling each other in the same VPC

Relevant content