Weird loadbalancing behavior from Classic ELB

We have 4x EC2 instances with httpd web server scattered across B and D zones. In random times of the day, one of the instances in Zone B would get none of the traffic and have its traffic rerouted the other EC2 instance in that zone.

The user-traffic would be like this:

1. Zone B, Instance 1 - 50%
2. Zone B, Instance 2 - 0%
3. Zone D, Instance 3 - 25%
4. Zone D, Instance 4 - 25%

I mentioned user-traffic above because the traffic from health checks during this period are equal across all machines and all of them are fine.

After 5 minutes it would return back to normal.

Any help would be appreciated.