macOS SMB uploads to Windows Server share hang for dozen of seconds

edit

Update 1: Default Congestion algorythm on MacOS is Cubic:

net.inet.tcp.newreno_sockets: 0
net.inet.tcp.background_sockets: 2
net.inet.tcp.cubic_sockets: 114
net.inet.tcp.use_newreno: 0
net.inet.tcp.cubic_tcp_friendliness: 0
net.inet.tcp.cubic_fast_convergence: 0
net.inet.tcp.cubic_use_minrtt: 0

Update 2:

If someone knows the following bullet points better, please correct me. All sources I have found, stated things sometimes differently so I tried to get it right connected.:

++++++++ can be discussed +++++++++++++++++++++

• Even Cubic uses the recovery algorithms of New Reno, it divers only in congestion avoidance state.
• an RTO causes cwnd=1 as long as all outstanding data is recovered
  • cwnd=1 causes the receiver side to go into delayed ACK
• the difference between the huge gap and the faster recovery times is SACK. A packet which causes a SACK have to send out immediately.
  • but due to the SACK problem on the server side, no SACK blocks can be reported anymore (See Sakes answer or the picture with the green rectangle below)

++++++++++++++++++++++++++++++++++++++++++++++++++

Original post:

Here comes my try of an answer: In the screenshot the followings things can be observed:

1. The server obviously supports only one SACK block
2. The client fills the gap correctly.
3. When the gap is closed the server stops advertising SACK. This is in this case an allowed behaviour, as it was not forbidden to discard packets which can´t be advertised in a SACK block
4. It can´t be told by evidence what has happened with the already transmited packets. But the server advertises them as lost packets (see second screenshot).
5. Therefor the client is still in Recovery mode and transmits the packets one by one. And obviously it is like @SYN-bit has pointed out some kind of Delayed ACK comes into play at the server side.

So why is it difefrent to Windows? I have three hypothesis in mind:

1. Windows sends out the data slower than the MacOS, so that no packet loss or packet Out of Order happens.
2. Windows does not use TCP-Timnestamping, so that the server advertises more SACK blocks.
3. Windows uses a different Recovery method

My analyis of the JPG's so far is:

• there is packet-loss, causing SACK block(s)
• when the first missing packet is received, the ACK jumps about 300k (frame 2267856)
• then the sender starts sending 1 MSS at a time (I think @Christian_R is right in pointing out the Reno behavior)
• the receiving side is using a delayed ACK mechanism, waiting for a second packet to do normal ACK
• since the sender only sends 1 MSS at a time, it times out after 100ms before sending the ACK
• after receiving the ACK, the sender sends 1 more MSS
• this process repeats until all gaps in the TCP stream have filled.

But I'd like to see the full SEQ, ACK, SACK etc values (ie see a pcap) to confirm this behavior.

I was privileged to take a look at the trace from an SMB level. Here are a few observations from the trace:

• In general, the server is somewhat sluggish when responding to write operations. The RTT is 312 microseconds. The service response for write operations is usually 5.x milliseconds
• The steady pattern of 5.x msec response time is rarely broken by responses as quick as 2 msec and the longest response time 25 seconds. The latter is caused by the famous retransmissions.
• The client creates the target file with options "read, write, append" on the file server. At the same time, the client permits sharing for read, write, and delete operations. As a result, the server has to prepare for lock management, even if the file is not really shared.
• The client uploads a file without specifying the size of the target file. As a result, the server has to expand the file on the fly. This opens the road to fragmentation on the file system.
• The client is not using SMB pipelining. Instead, it sends the next data block only, after the last operation completed.

Unless the transfer is stopped by the packet loss (as analyzed by Sake, Christian and Phil) data is transmitted at approx. 750 MBit/sec. This does not leave a lot of room for improvement for a 1 GBit link.

A few things to check on the server side (if possible, or still necessary)

• Check, if the variations in response times are related to operations from another client (more Wireshark)
• Check disk fragmentation
• Check the hard disk health status
• Run onboard diagnostics to identify other bottlenecks on the server side (CPU, memory etc.)
• Check the server's RAID architecture. RAID-5 is probably not the best choice for applications involving lots of write operations.

Thank you Tom, for the interesting case