Ask Your Question
0

TCP SYN replied immediately by RST after successful session

asked 2018-02-26 08:30:48 +0000

this post is marked as community wiki

This post is a wiki. Anyone with karma >750 is welcome to improve it.

We are trying to deploy a new TCP client to production, after it has been successful working against a test server. Only with production server, it succeeds 1 or 2 TCP sessions, but after that it always fails on connection attempt (SYN) which does not reach our application. The client runs on Linux, and attempts a connection every 15 minutes. The server OS is Windows Server 2008 R2, and our application on it is a Windows Service. Aside from this client, about 400 other clients are communicating with this server with frequencies ranging between every few seconds to every few minutes. This client always attempts to connect to server port 4770, the other clients try server ports in the range 4770-4772 but each client always connects to the same server port. Server IP is 192.168.33.99, client IP is 10.211.12.236. Please see server-side capture here. Is the last 'successful' stream not terminated correctly? Who's to 'blame' for the fault - server or client?

edit retag flag offensive close merge delete

Comments

You can post the file at cloudshark or any plain file sharing service and edit your Question with a login-free link to it.

From the contents I guess you've captured the attempts on the client (linux) side. Can you also capture at the server in parallel? Some device between the two may have messed up the traffic. The time gap between the end of the session from 50123 and the SYN of the session from 50124 is so large that I don't think they are related at TCP level even if the box-in-the middle would change the client side port to the same one for both (which would normally be my first suspicion as it often happens). However, I can see that packets get lost between the client and the server, as the server has retransmitted your packet #871 once again as #882 although the ACK to the ...(more)

sindy gravatar imagesindy ( 2018-02-26 09:36:54 +0000 )edit

Thanks sindy, I have edited my question and added a link to the full capture. Sorry for the confusion - capture is server side.

inigomontoya gravatar imageinigomontoya ( 2018-02-26 11:56:13 +0000 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-02-27 10:27:55 +0000

nik gravatar image

updated 2018-03-01 17:42:05 +0000

The second TCP connection ending around frame 871 should be considered complete. If you see FIN packets on both sides the connection has ended from the standpoint of either side.

Correction1: I don't know why I thought the source port is not moving. My bad.
You have a client-server pair that is re-using the TCP ports (specifically the client is re-using the source port). This too should not be a problem because of the long delay of 15 minutes between the last FIN sequence and the new SYN sequence.

Looking at the SYN packets of the first two successful connection sessions and comparing against the SYNs that lead to RST responses, there seems to be no significant difference that would cause them to invoke RSTs. This means, the client is probably not responsible for these RSTs.

As recorded on the server, it is returning RSTs to inbound SYNs from the linux client. Based on previous observations here, it is more probable that the server is responsible for the RSTs. The cause should not be external (other TCP communications/ packets) because we have a server side capture.

Correction1: re-use is out of question because source port is moving.
While, a "TCP connection re-use" issue does not appear to be a likely reason for this behavior, can you (nevertheless) try making the client rotate source ports from a bunch of port numbers (say 3 or 10)? If you get a reproduction with that change it will eliminate the re-use related issues completely.

As an aside, have you investigated the reason for the re-transmits in the first two TCP sessions?
Update1: There are also lost frames in the communication (example: between frames 757 and 759 a bunch of client sourced frames are missing from the capture).

One more aspect to check is the other communications that Windows server is handling. If there are too many connections on going that could load its TCP stack. But, a consistent reproduction with failures starting at the 3rd connection on 15 minute intervals does not seem to suggest a "3rd party" interference.

Update1: Based on your comment about 400 other clients communicating with this server at connection rates within-seconds there are two aspects to check on the TCP Server system.

  1. Are all connections closing correctly on the server? If there are packet losses some connections could be holding on timeouts. This might prevent future connections setting up on the server (too many open connections).
  2. Can the server sustain this load of connections? (Is this an Enterprise Edition, how much RAM and compute power, etc.)

Can you access the server logs / events to check if there are any errors being logged for TCP communications?
Is the server behaving correctly for the other 400 clients? or, are all of these showing up similar issues?

That the server PCAP sample shows missed inbound frames also suggests that the server is being over-run by communications on its interface.


segue on use of Locally Administered MAC addresses.
It appears from your ... (more)

edit flag offensive delete link more

Comments

From what I see, the client is rotating source ports starting with 50122 and incrementing by 1 on each new connection attempt. We have not investigated the packet losses, because we were unable to install a sniffer on the client. As for load, there are about 400 other clients actively communicating with this server in higher frequencies - ranging between every few seconds to every few minutes. I am not sure if that implies a significant load on the server's TCP stack.

inigomontoya gravatar imageinigomontoya ( 2018-02-28 15:30:27 +0000 )edit

Thanks for your elaborated response, I will check these points

inigomontoya gravatar imageinigomontoya ( 2018-03-13 09:47:00 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2018-02-26 08:30:48 +0000

Seen: 171 times

Last updated: Mar 01