Hi,

I'm trying to figure out a problem where I'm getting multiple socket exceptions on client machines on the network. Clients always connect to the server, send some data and the server always sends some data back to every client. I've run a prolonged capture and I'm seeing that when the problem occurs, the server seems to be sending the data back to the client, but almost immediately after that the server sends an RST+ACK packet, as shown below:

No.     Time        Source                Destination           Protocol Length Info
  57081 0.000000    10.41.0.9             10.41.1.100           TCP      62     [TCP Port numbers reused] 1224 > 1234 [SYN] Seq=0 Win=32768 Len=0 MSS=1460 SACK_PERM=1
  57082 0.000039    10.41.1.100           10.41.0.9             TCP      62     1234 > 1224 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 SACK_PERM=1
  57083 0.003693    10.41.0.9             10.41.1.100           TCP      60     1224 > 1234 [ACK] Seq=1 Ack=1 Win=33580 Len=0
  57084 0.031041    10.41.0.9             10.41.1.100           TCP      135    1224 > 1234 [PSH, ACK] Seq=1 Ack=1 Win=33580 Len=81
  57087 0.113171    10.41.1.100           10.41.0.9             TCP      54     1234 > 1224 [ACK] Seq=1 Ack=82 Win=65454 Len=0
  57088 0.069353    10.41.1.100           10.41.0.9             TCP      74     1234 > 1224 [PSH, ACK] Seq=1 Ack=82 Win=65454 Len=20
  57095 0.104433    10.41.1.100           10.41.0.9             TCP      54     1234 > 1224 [RST, ACK] Seq=21 Ack=82 Win=0 Len=0

A more detailed log is available under http://winger.pl/userfiles/err.txt .

Does anyone have any suggestions of to what might be causing the [RST, ACK] packets to be sent? The clients seem to be getting the RST/ACK (causing them to throw a socket exception) but I'm not sure they are getting the data. There are no firewalls or routers between the host and the client, any kind of firewall/AV software has been disabled on the system running the host application. Also, the same host application is being used on numerous other locations and I havent' seen this problem anywhere else.

Kind Regards, Winger

asked 05 Aug '11, 10:44

Winger's gravatar image

Winger
6114
accept rate: 0%


TCP RST packets should not be seen normally, one exception is when a Microsoft client closes an SSL session, then you might see "normal" TCP RST packets.

The fact that your first show frame (#57081) has "[TCP Port numbers reused]" in the info column means that Wireshark has seen another TCP session with the same IP-addresses and port numbers before. I think you need to focus on that. Filter on the port numbers and check how long these two (different) TCP sessions were apart, and then check whether the TIME_WAIT timer on 10.41.1.100 might be longer than that.

If so, you need to look into the amount of sessions that you need to process and tune the port range that can be used and the TCP timers accordingly (or even start using multiple IP addresses for making these connections).

It's also a good idea to compare all bad sessions with the good sessions. What do they have in common?

link

answered 07 Aug '11, 07:10

SYN-bit's gravatar image

SYN-bit ♦♦
15.0k848219
accept rate: 19%

Thanks for the tips - they were helpful (especially the first sentence, as after some googling around I got a feeling that RST is pretty standard). I was able to figure out that the host app does not handle closing of the connections properly (they were closed by the clients but they lingered on from the server's perspective) - every single connection ended with RST, once this was fixed the issue has disappeared.

(08 Aug '11, 09:44) Winger

Glad to hear you were able to solve your issue!

(08 Aug '11, 10:09) SYN-bit ♦♦

It is not uncommon for servers to terminate TCP connections by sending a RST, simply because it is faster than FIN/ACK/FIN/ACK, and there is no TIME_WAIT status afterwards. So far I interpret your trace like this:

  • Packets 57081-57083: Three Way Handshake
  • Packet 57084: Client requests something from the server and uses PSH to tell it to go ahead and process
  • Packet 57087: the server acknowledges the request, without replying with any content (len=0, so no payload in that packet)
  • Packet 57088: the server sends the payload, which took 0.069 seconds to generate (determinted by the delta from the acknowledge in 57087)
  • Packet 57095: the server closes the connection with a RST packet. Maybe because it knows that its work is done, or maybe because there was no further client request within a certain amount of time.

So, keep in mind that a reset (RST) packet unfortunately doesn't mean anything went wrong. It could be a "normal" connection termination. This kind of termination is often used by Microsoft products like Internet Explorer, Outlook, Exchange Server etc.

link

answered 05 Aug '11, 17:40

Jasper's gravatar image

Jasper ♦
16.1k338212
accept rate: 17%

Hi,

Thanks for the answer. I suppose I should have added that the server application never terminates the connection in that setup - it's always the client doing this after it receives and procesess the data; therefore, in the above example, I would have received FIN/ACK from the client (after packet 57088) and this would be followed by ACK and RST/ACK sent back to the client. The code of the server application simply doesn't perform the closing of the connection. Hence, I'm puzzled why this is occuring and I'm trying to understand the reasons behind it.

The problem only occurs for approx. 5% of all the transactions taking place.

Kind Regards, Winger

(06 Aug '11, 03:31) Winger

(converted your "answer" to a "comment", please see the FAQ)

(07 Aug '11, 06:58) SYN-bit ♦♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×75
×73
×48
×4
×2

Asked: 05 Aug '11, 10:44

Seen: 47,493 times

Last updated: 08 Aug '11, 10:09

powered by OSQA