RST packets sent by both client and server during file transfer

asked 2018-07-11 15:00:01 +0000

FB
3 ●1 ●1 ●3

updated 2018-07-11 19:57:32 +0000

I am trying to troubleshoot why a file upload from a client to Linux server fails in our network. From the workstation, I want to upload a file to the server, so I run the command

scp scan-20170710.pdf [email protected]:/tmp/

After entering the password, I get the message: "packet_write_wait: Connection to 10.11.12.26 port 22: Broken pipe"

I tried capturing the communication between the client and the server. There are no hops between the two, only two switches and an AP. I am linking the server side capture and the client side capture.

The client and the server seem to communicate perfectly fine until suddenly they start sending RST packets to each other. What seems also odd to me is the fact that each RST packet can be seen on the receiving side but not on the sending side.

The file that I am uploading from the workstation is 1.8 MB in size. If I try uploading a smaller file ~100 kB, the issue never occurs. In addition, this error never occurs if I download a file bigger than 1 MB from the server.

I have been looking online for potential solutions, but everything seems to be linked to the timeout time of the session being too short, which I don't think is the problem in this case because the error message pops up less than a second after I enter the password.

Has anyone ever seen a similar case?

edit retag flag offensive close merge delete

Comments

Links require permission to access the packet capture files. It's difficult to see what is going on when we can't see the data set you have. Suggest to remove permissions on the files.

Bob Jones ( 2018-07-11 16:10:44 +0000 )edit

I apologize. Please, try now.

FB ( 2018-07-11 16:34:41 +0000 )edit

Still asks for permission...

NJL ( 2018-07-11 17:39:57 +0000 )edit

Didn't realize I was on closed drive. It should work now.

FB ( 2018-07-11 19:04:45 +0000 )edit

There are no RST packet in these captures - Are we looking at the right data ?

mrEEde ( 2018-07-11 19:44:24 +0000 )edit

see more comments

answered 2018-07-12 03:34:13 +0000

Jim Aragon
7512 ●7 ●71 ●128 Fairfield, California

What seems also odd to me is the fact that each RST packet can be seen on the receiving side but not on the sending side.

The RST packets claiming to be from the client and the RST packets claiming to be from the server start at approximately the same time.
The RST packets claiming to be from the client are seen only on the server, and the RST packets claiming to be from the server are seen only on the client.
There are the same number of RSTs in each direction.
With one exception, the IP TTLs on the RST packets are different from the IP TTLs on all the other packets. All the non-RST packets have a TTL of 64, except for packet 1 in the server trace, which has a TTL of 63. All the RST packets have a TTL of 250, except for packet 114 in the client trace and packet 107 in the server trace, which have a TTL of 64.

RST packets are not supposed to be responded to. If one system receives a RST, it should just silently tear down the TCP connection. It should not send any more packets to the other system.

I'd say that the client and the server are not sending RSTs to each other. Instead, some device in the middle is aborting the TCP session by sending RSTs in both directions. It's spoofing the server's address in RST packets going to the client, and it's spoofing the client's address in RST packets going to the server. It has to do that in order to get the client and server to accept and take action based on the RSTs. They will not tear down an established TCP connection is response to a RST packet from a third device that is not part of that connection, so the client has to believe that the RST is from the server and the server has to believe that the RST is from the client.

Look for some other device in between the client and the server, probably one whose operating system has a default TTL of 250. If there's really nothing between the client and the server except the Access Point and two switches, then it's one of those. Are you sure that there is nothing else between the client and server? Maybe some security device that is in transparent mode?

edit flag offensive delete link

Comments

Thank you for in depth analysis. Since I posted this question, I was informed that one of the devices in between is not actually a switch but a Meraki router.

We tried uploading the same type of file connecting directly through the switch in front of the server (i.e. not going through the router) and the upload worked fine. Connecting to your analysis, I would say that the router must be spoofing the endpoints and terminating the communication. I am not sure why, though. I will post again once I figure it out.

As a follow-up question, is it normal that only the first packet has a TTL of 63 and all the others have a TTL of 64 if there is a router is the middle? I was expecting to see a TTL of 63 on all the packets.

Thank you again for your help.

FB ( 2018-07-13 12:32:49 +0000 )edit

The key is the "receiver sees it, but the sender didn't send it" evidence in the respective capture files. In EVERY instance in which I've seen that behavior over the years, it has been an intermediate device - a load balancer, intrusion detection system (IDS), or firewall - that forged the RSTs in both directions.

I've even seen this in the consumer Internet world - one large ISP decided that any transactions lasting longer than X seconds "just had to be" illicit file sharing, so they configured their security devices to simply throw RSTs in both directions...which wasn't good for our customers replicating large databases from the cloud while working from home.

wesmorgan1 ( 2018-07-30 05:11:26 +0000 )edit

add a comment

RST packets sent by both client and server during file transfer

Comments

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

RST packets sent by both client and server during file transfer edit

Comments

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

RST packets sent by both client and server during file transfer