Ask Your Question
0

RST packets sent by both client and server during file transfer

asked 2018-07-11 15:00:01 +0000

FB gravatar image

updated 2018-07-11 19:57:32 +0000

I am trying to troubleshoot why a file upload from a client to Linux server fails in our network. From the workstation, I want to upload a file to the server, so I run the command

scp scan-20170710.pdf [email protected]:/tmp/

After entering the password, I get the message: "packet_write_wait: Connection to 10.11.12.26 port 22: Broken pipe"

I tried capturing the communication between the client and the server. There are no hops between the two, only two switches and an AP. I am linking the server side capture and the client side capture.

The client and the server seem to communicate perfectly fine until suddenly they start sending RST packets to each other. What seems also odd to me is the fact that each RST packet can be seen on the receiving side but not on the sending side.

The file that I am uploading from the workstation is 1.8 MB in size. If I try uploading a smaller file ~100 kB, the issue never occurs. In addition, this error never occurs if I download a file bigger than 1 MB from the server.

I have been looking online for potential solutions, but everything seems to be linked to the timeout time of the session being too short, which I don't think is the problem in this case because the error message pops up less than a second after I enter the password.

Has anyone ever seen a similar case?

edit retag flag offensive close merge delete

Comments

Links require permission to access the packet capture files. It's difficult to see what is going on when we can't see the data set you have. Suggest to remove permissions on the files.

Bob Jones gravatar imageBob Jones ( 2018-07-11 16:10:44 +0000 )edit

I apologize. Please, try now.

FB gravatar imageFB ( 2018-07-11 16:34:41 +0000 )edit

Still asks for permission...

NJL gravatar imageNJL ( 2018-07-11 17:39:57 +0000 )edit

Didn't realize I was on closed drive. It should work now.

FB gravatar imageFB ( 2018-07-11 19:04:45 +0000 )edit

There are no RST packet in these captures - Are we looking at the right data ?

mrEEde gravatar imagemrEEde ( 2018-07-11 19:44:24 +0000 )edit

2 Answers

Sort by ยป oldest newest most voted
2

answered 2018-07-12 03:34:13 +0000

Jim Aragon gravatar image
What seems also odd to me is the fact that each RST packet can be seen on the receiving side but not on the sending side.
  1. The RST packets claiming to be from the client and the RST packets claiming to be from the server start at approximately the same time.
  2. The RST packets claiming to be from the client are seen only on the server, and the RST packets claiming to be from the server are seen only on the client.
  3. There are the same number of RSTs in each direction.
  4. With one exception, the IP TTLs on the RST packets are different from the IP TTLs on all the other packets. All the non-RST packets have a TTL of 64, except for packet 1 in the server trace, which has a TTL of 63. All the RST packets have a TTL of 250, except for packet 114 in the client trace and packet 107 in the server trace, which have a TTL of 64.

RST packets are not supposed to be responded to. If one system receives a RST, it should just silently tear down the TCP connection. It should not send any more packets to the other system.

I'd say that the client and the server are not sending RSTs to each other. Instead, some device in the middle is aborting the TCP session by sending RSTs in both directions. It's spoofing the server's address in RST packets going to the client, and it's spoofing the client's address in RST packets going to the server. It has to do that in order to get the client and server to accept and take action based on the RSTs. They will not tear down an established TCP connection is response to a RST packet from a third device that is not part of that connection, so the client has to believe that the RST is from the server and the server has to believe that the RST is from the client.

Look for some other device in between the client and the server, probably one whose operating system has a default TTL of 250. If there's really nothing between the client and the server except the Access Point and two switches, then it's one of those. Are you sure that there is nothing else between the client and server? Maybe some security device that is in transparent mode?

edit flag offensive delete link more

Comments

Thank you for in depth analysis. Since I posted this question, I was informed that one of the devices in between is not actually a switch but a Meraki router.

We tried uploading the same type of file connecting directly through the switch in front of the server (i.e. not going through the router) and the upload worked fine. Connecting to your analysis, I would say that the router must be spoofing the endpoints and terminating the communication. I am not sure why, though. I will post again once I figure it out.

As a follow-up question, is it normal that only the first packet has a TTL of 63 and all the others have a TTL of 64 if there is a router is the middle? I was expecting to see a TTL of 63 on all the packets.

Thank you again for your help.

FB gravatar imageFB ( 2018-07-13 12:32:49 +0000 )edit

The key is the "receiver sees it, but the sender didn't send it" evidence in the respective capture files. In EVERY instance in which I've seen that behavior over the years, it has been an intermediate device - a load balancer, intrusion detection system (IDS), or firewall - that forged the RSTs in both directions.

I've even seen this in the consumer Internet world - one large ISP decided that any transactions lasting longer than X seconds "just had to be" illicit file sharing, so they configured their security devices to simply throw RSTs in both directions...which wasn't good for our customers replicating large databases from the cloud while working from home.

wesmorgan1 gravatar imagewesmorgan1 ( 2018-07-30 05:11:26 +0000 )edit
0

answered 2024-10-25 10:42:53 +0000

SYN-bit gravatar image

I stumbled on this old question because I was searching evidence that Meraki has a default TTL of 250. It seems to be the case...

Regarding your question:

As a follow-up question, is it normal that only the first packet has a TTL of 63 and all the others have a TTL of 64 if there is a router is the middle? I was expecting to see a TTL of 63 on all the packets.

Yes, normally you would indeed see all packets that are routed over one hop decrease the TTL, so in your packet captures, you would see a TTL of 63 for packets coming in from the other side of the Meraki router. But apparently, the Meraki works a bit differently, and it also does not seem to like the TCP seqments with sequence numbers starting with 48502, as these frames are forwarded to the server as a RST packet (look at the sequence numbers) while at the client these packets are answered with a RST.

The final RST is an actual RST of the client, after it received an ACK, after the connection was already reset.

Based on the timings in the pcap files, I assume the packet captures were made on the client (Mac) and the server (VMware VM)? And were you ever able to find the reason for why the Meraki router killed large file transfers?

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2018-07-11 15:00:01 +0000

Seen: 3,067 times

Last updated: Oct 25