Can you explain this TCP sequence

edit

The checksum can be wrong, but it also can be due to offloading at least in the trace with the huge frames.

I thought about that but how to explain such selectiveness - only original segment with sequence # 3020828 and all its subsequent retransmissions were dropped, while all of them had different checksums (because of different TCP ACK fields). At the same time none of pure ACKs from the same source was dropped.

I guess, that the packets get stuck in the queue of the stack, as we can see that the stack is under stress due the advertised RWin of 4096. So I agree with Vladimir slowing down the traffic could be tried.

Thanks, Maynard. This is a good hint - unfortunately, the original registrations are complete (possible that TraceWrangler truncates the payloads).

From Wireshark, I enabled checksum verification on the receiver, and the checksum are indeed correct, according to Wireshark.

Additional note: yes, there is Checksum Offloading enabled on both side, so checksum can only be verified on the registration side showing packet receive.

Thanks, @Christian_R and @Packet_vlad. I see all the points about reducing/slowing down the traffic by tuning the TCP window scaling factor and the TCP buffer size. And they are indeed very pertinent. But both seems palliative actions: I would like to understand which is the root cause of the TCP behavior here and why the back pressure is not activated on the sender side.

Hi Luca, I assumed reducing Window Scale factor as a measure to improve our observation ability. If you reduce WS factor, you'll be able to see what is actual (or at least more granular) RWIN size at the moment, not the one aligned to 4096-Bytes steps. And this would give us a possibility to do further analysis. Of course if it reveals us RWIN big enough to store a couple of MSS, the hypothesis is wrong and we'll search elsewhere.

If we are right, and the systems are just under stress, then you can either slow down connections or making the systems itsself faster (checking and tuning CPU load, system load....and so on; It can be a complex task)

Thanks guys. We will retry with the TCP scaling and window size changed, then. What remains unclear to me is why in such stress condition, the TCP machinery fails in triggering the (so much needed) congestion control.

I talk about the snd.pcapng

It is triggered. The congestion controls are in play:

1. We see a lot of Zero Windows

2. We see also small advertised windows.

3. Congestion Avoidance comes into play with the Retransmission (#58376 for the SrcPort 64613 and # 91696 for the SrcPort 14004), as they are Retransmission of time (RTO) they will reduce the sending window dramatically but first at this point, but your RTT is small, too. So BDP might be still high.

I understand the receiver application is temporarily busy and not consuming data from the receive window, while it is perfectly able to send data out. We know this could be normal for such kind of application at startup. However, either of the two must be true: 1) the receive buffer is eventually full: a Zero Window must be acknowledged. 2) the receive buffer is not full: the incoming retransmission packet must be (at least partially) accepted and acknowledged.

In the registration, each retransmission of len 1460 is not acknowledged, yet a receive window of 4096 is advertised. This is the only thing that still sounds inconsistent to me.

The receiver application is a Java application, using java.nio and non-blocking sockets. An incoming request could easily take the application busy for a while, producing a significant amount of data out in result. So, no surprise its receive window shrinks from time to time -- but if it's full, it must go to 0 and not let the sender silently think the packet is lost.