Mystery Intermittent Connection Refused Error
Having one of the mysterious connection refused which are intermittent (5-6 times a day for 24x7 application).
Connection Setup:
Source - Fireall - Load Balancer (VIP) - Target
Source - Linux servers (3 nodes) Target - Windows servers (2 nodes) Load Balancer - Cisco ACE
There is no significant load on servers but every 1 minute an application (web service) call is made by source application to get application data.
We have network capaturea at Load Balancer and Target Servers. Here is what we see.
- Source sending SYN to VIP and getting back RESET.
- We don’t see this SYN packet at both Target servers.
- Both Target servers getting Pings from Load Balancer at every 5-10 seconds indicating servers are up.
- More than 90% of sockets/thread available on Target servers.
- We’ve performed load test several times and this connection refused error never produced during load test.
- Very next request from same source and target will get success.
Firewall team thoeriticallt denies that it can be Firewall Issue. Network team said Load Balancer cannot send RESET and it would come from an application. We do not see any activity at server to suggest it can be server issue.
Can servers reject connection and not show-up on Wireshark?
What can be this problem and how to address this problems?