Can client control whether server sets PSH flag on every packet?

asked 2019-08-16 05:49:59 +0000

hcaandersen
16 ●1 ●1

I'm comparing the behaviour of two pieces of SFTP client software. They're doing the same SFTP download, but one is 15 times faster than the other. Some analysis has led me to believe that the problem may be related to PSH flags. Both are downloading data in chunks of 32,768 bytes. In both cases the data is transferred back from the server in packets of size 1,360 bytes. In one, every packet has the PSH flag set. In the other, only the final packet of a chunk has the PSH flag set.

My question is: what, on the client-side, is causing this difference in use of PSH flags?

edit retag flag offensive close merge delete

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2019-09-23 00:52:27 +0000

hcaandersen
16 ●1 ●1

It turned out that the performance issue had nothing to do with the PSH flag. The problem was that the SFTP server had a latency of 300ms for every read-block request. The faster SFTP client masked this delay by queuing up dozens of read-requests and processing the responses asynchronously, whereas the slower one would proceed synchronously, waiting for each response before sending the next one. It was not obvious that the faster client was performing asynchronously because requests were encrypted so I couldn't see which response corresponded to which request and therefore couldn't trace individual block-reads.

edit flag offensive delete link

Comments

Another member asked me for the names of the clients.

They were WinSCP (a Windows application) and edtFTPnet/PRO (a .NET library).

WinSCP uses the Putty code-base, which queues read requests (see xfer_download_queue) up to a maximum total buffer size of 1MB. OpenSSH queues SFTP read requests (see here) up to a maximum of 64 requests.

edtFTPnet/PRO up until version 9.5.1 didn't queue read requests, instead performing them sequentially. In other words, a read request would be sent, the client would wait for a response, the response would arrive and then the next read request would be sent. An algorithm similar to that of OpenSSH was implemented and this resulted in a speed-up of around 10 times against this server. Speed-ups were observed with other servers as well, though there weren't nearly as great.

edtFTPnet/PRO didn't normally perform so poorly since SFTP servers ...(more)

hcaandersen ( 2019-09-24 23:09:36 +0000 )edit

add a comment

answered 2019-08-16 06:22:01 +0000

SYN-bit

18528 ●9 ●343 ●255 https://SYN-b.it

The Push Flag is used by the sending side to mark that it is end of a data chunk (Ex. end of data header or data body being sent). This tells the TCP layer on the receiving side to flush the received data to the receiving application without waiting for the receive buffer to be filled. The setting of the Push Flag is usually not controlled by the sending application, but by the sending TCP layer. Most modern TCP/IP stacks set the PSH bit at the end of the buffer supplied to send() . The Push Flag helps the receiving side to optimize throughput by bundling data into logical chunks.

(from : http://smallvoid.com/article/winnt-tcp-push-flag.html)

My guess is that one of the applications calculates the MSS itself to send blocks of data of MSS bytes to the TCP layer (in which case I find it strange that it also uses 32768 byte blocks, it could just keep sending segments of MSS length) and the other just sends blocks of 32768 bytes to the TCP layer. In the first case all packets will have the PSH bit set and in the second case only the last segment of the 32768 byte block will have the PSH bit set.

Which application was quicker? Are you able to share the capture files on a public fileservice like Dropbox, GoogleDrive, OneDrive etc? If not, which SFTP clients are you testing?

edit flag offensive delete link

Comments

Maybe there is a device in between (LB,FW, ...) which sets a PSH bit to every tcp packet, but this is only guessing, as I don't know where you took the trace.

Where did you take the trace?

Christian_R ( 2019-08-16 21:37:16 +0000 )edit

Thank you so much for replying. I can't share the raw Wireshark logs for security reasons, but I've exported snippets of the summaries to text files and shared those:

These are recorded on the SFTP client side, which is all we have access to.

I've replaced the actual IP addresses with SFTP_CLIENT_1, SFTP_CLIENT_2 and HTTP_PROXY. SFTP_CLIENT_1 is the slow one. The SFTP server is behind an HTTP proxy, so I've replaced the remote hosts's IP address with HTTP_PROXY to indicate this. The proxy and the server are the same in both cases, so it seems apparent that it's something that SFTP_CLIENT_1 is doing wrong that's causing the problem.

Things to notice:

All of the packets sent to SFTP_CLIENT_1 have the PSH flag set (which motivated my question)
HTTP_PROXY takes >250ms to reply to the Len=68 packets (which are ...

(more)

hcaandersen ( 2019-08-19 01:40:14 +0000 )edit

Thank you for the snippets. I was expecting the client to set the PSH flag, but somehow the SFTP_CLIENT_1 manages to induce HTTP_PROXY to set the PSH flag. So that makes this even more interesting, as the HTTP_PROXY and the SFTP_SERVER behind it are the same in both scenarios.

I understand the security concerns. Would it be possible for you to use Tracewrangler to anonymize the pcaps? You can alter the mac- and IP addresses and if needed, also replace all payload above TCP level (although if possible, it would be nice to keep this payload)? Will that be sufficient to be bale to share the anonymized pcaps? I would really like to look at the whole transfer at the TCP level. But in the end, I think the cause of the difference in behavior must be something in a difference in the SSH settings of the SFTP connection.

Some ...(more)

SYN-bit ( 2019-08-19 08:54:36 +0000 )edit

I'm reluctant to even ask our customer if we can send the traces to a third party, so I'll see how far we can get without doing so.

It turns out that I'd misunderstood the network topography and that the proxy is actually on our customer's network, which means that I'm going to be able to get traces from the proxy. I should get those in a few hours, so I'll update this thread again then.

This, of course, also means that I can ask our customer to try to connect through the proxy to another server (ours), and perhaps to bypass the proxy, like you suggested. I'll come back with those results when they arrive also.

Thanks again!

hcaandersen ( 2019-08-20 00:28:50 +0000 )edit

No problem (regarding the traces), I understand the concern. Good luck with your next tests, I hope they will give some insight in the behavior of these clients (and the proxy). Is the proxy a virtual IP on a loadbalancer (so operating at L4) or is it a full application layer proxy that terminates the SSH connection and sets up a new SSH connection on the backend?

SYN-bit ( 2019-08-20 05:53:39 +0000 )edit

see more comments