Can you explain this TCP sequence

edit

The checksum can be wrong, but it also can be due to offloading at least in the trace with the huge frames.

I thought about that but how to explain such selectiveness - only original segment with sequence # 3020828 and all its subsequent retransmissions were dropped, while all of them had different checksums (because of different TCP ACK fields). At the same time none of pure ACKs from the same source was dropped.

I guess, that the packets get stuck in the queue of the stack, as we can see that the stack is under stress due the advertised RWin of 4096. So I agree with Vladimir slowing down the traffic could be tried.

Thanks, Maynard. This is a good hint - unfortunately, the original registrations are complete (possible that TraceWrangler truncates the payloads).

From Wireshark, I enabled checksum verification on the receiver, and the checksum are indeed correct, according to Wireshark.

Additional note: yes, there is Checksum Offloading enabled on both side, so checksum can only be verified on the registration side showing packet receive.

Thanks, @Christian_R and @Packet_vlad. I see all the points about reducing/slowing down the traffic by tuning the TCP window scaling factor and the TCP buffer size. And they are indeed very pertinent. But both seems palliative actions: I would like to understand which is the root cause of the TCP behavior here and why the back pressure is not activated on the sender side.

Hi Luca, I assumed reducing Window Scale factor as a measure to improve our observation ability. If you reduce WS factor, you'll be able to see what is actual (or at least more granular) RWIN size at the moment, not the one aligned to 4096-Bytes steps. And this would give us a possibility to do further analysis. Of course if it reveals us RWIN big enough to store a couple of MSS, the hypothesis is wrong and we'll search elsewhere.

If we are right, and the systems are just under stress, then you can either slow down connections or making the systems itsself faster (checking and tuning CPU load, system load....and so on; It can be a complex task)

Thanks guys. We will retry with the TCP scaling and window size changed, then. What remains unclear to me is why in such stress condition, the TCP machinery fails in triggering the (so much needed) congestion control.

I talk about the snd.pcapng

It is triggered. The congestion controls are in play:

1. We see a lot of Zero Windows

2. We see also small advertised windows.

3. Congestion Avoidance comes into play with the Retransmission (#58376 for the SrcPort 64613 and # 91696 for the SrcPort 14004), as they are Retransmission of time (RTO) they will reduce the sending window dramatically but first at this point, but your RTT is small, too. So BDP might be still high.

I understand the receiver application is temporarily busy and not consuming data from the receive window, while it is perfectly able to send data out. We know this could be normal for such kind of application at startup. However, either of the two must be true: 1) the receive buffer is eventually full: a Zero Window must be acknowledged. 2) the receive buffer is not full: the incoming retransmission packet must be (at least partially) accepted and acknowledged.

In the registration, each retransmission of len 1460 is not acknowledged, yet a receive window of 4096 is advertised. This is the only thing that still sounds inconsistent to me.

The receiver application is a Java application, using java.nio and non-blocking sockets. An incoming request could easily take the application busy for a while, producing a significant amount of data out in result. So, no surprise its receive window shrinks from time to time -- but if it's full, it must go to 0 and not let the sender silently think the packet is lost.

And this is exactly the point I'm trying to investigate :) Let's say receive buffer has 1000 Bytes free, it is not full. How are you going to report it IF your Window scaling Factor is 4096? Even if you set 1 as window size - it should be scaled and interpreted by sender as "Free 4096 Bytes". You can't decrease Scaling factor during connection, because it is negotiated in 3-way handshake and is constant throughout all connection.

Honestly I do not know how TCP has to behave in this sutiation. Set to 1? It's no true, the buffer is smaller. Set to 0? Not true either... Please guys point me where to read about such case.

I've also never seen such thing as "Partial ACK", when receiver ACKs less then one packet sent to it although it is not forbidden as I know.

What will be if the packet is silently dropped by one of the internal queues? Just a thought.... have no evidence for this.

Not sure whether it could be pertinent or it just adds another variable into the picture, but... The issue seems to be empirically solved by enabling jumbo frames on the sender box. Empirically means it used to reproduce 8 out of 10 tries before; it does not replicate (yet) now with jumbo frames on. Does this makes any sense? Clearly, the level of congestion both sides is identical, since neither the data volumes nor the application behavior changed.

Jumbo Frames will fit better with your scaling factor, as Vladimir has mentioned about the size of 4096. And of course do Jumbo Frames take away some load from your system. So yes it can make sense especially with your high Window Size values and your small RTT.

But you must ensure that every box supports Jumbo Frames even the receiver. Otherwise Jumbo Frames will not be used. https://crnetpackets.com/2016/01/27/t...

Could you please share this new trace for education purpose? I'm so interested to see current behavior.

Yes, @Christian_R. It turned out jumbo frames were enabled everywhere, but on the sender box: so enabling it on the sender box made them flow all the way end-to-end.

@Packet_vlad, sure: I will do this first thing when I get my hands on them.

Many thanks, @JimAragon. So we're basically stuck at the beginning: we have a recv box dropping 1460-byte packets, yet acknowledging a 4k recv window.

Jim, great explanation, thanks! I think I'll try to perform some experiments to observe how it happens in real systems. Luca, maybe this is a good idea to check some OS counters regarding dropped or error packets at the time when issue is happening.

Hello, guys. Thanks for you interest so far: this is getting pretty catchy.

So I have another registration, washed with TraceWrangler, for the connection NOT working: https://drive.google.com/open?id=1qYp... This time the recv window gets down to 2k and there it remains until the RESET: see 14012 with packet seq #2937795, when the retransmissions start.

I also have the registration with Jumbo frames on, washed with TraceWrangler: https://drive.google.com/open?id=1PIA... This time, same applications, same data volumes, the connection stays perfectly alive. The receiver is eventually able to send Zero Window, see 17475 which seems to be first occurrence.

Quite interesting. For the Jumbo Frames. We see packet loss and Fast Retransmission and RTO. Maybe something is wrong with your driver. Maybe there is a driver issue with the Offloading functions. Not so impossible. Do you use the newest Version of the drivers?

And also it is interesting what OS and version is this..

@Packet_vlad: Yes that would be interesting. And what network cards and drivers are used.

What is interesting to me is the fact once you've decreased WS factor to 2048, the host started to report exactly 1(=2048) instead of 1(=4096). Could you please do couple steps further and lower it to, say, 256 or even 128? You have largest "Bytes in flight" number of 640k, so WS factor of 16 would be enough to handle this load.

In "Jumbo-trace" the receiver reported 3(=6144) RWIN size which is less than 1MSS (=8946) so the sender was able to create packet which fits perfectly into RWIN (packets 17472 + 17473) which caused Zero Window (this is good correct behavior).

Sender is a Windows desktop, while Receiver is on RedHat 6, June update, Kernel 2.6.32-754.2.1

Hello, guys. So basically, the issue is systematically reproducible even with simulator applications, under the following conditions: 1) the Sender running on Windows box 2) then Receiver running on Red Hat 6 box (we are on Kernel 2.6.32-754.2.1) 3) the Windows Scaling Factor of the Receiver to 4096 4) the Receiver uses SO_RCVBUF option on socket 5) Congestion is generated from the Sender to the Receiver, in such a way the Receiver is slow but keeps reading data.

With the above, after enough time the connection gets stuck in retransmissions, while the Receiver fails in acknowledging a Zero Window.

It seems certain conditions or combination of parameters are preventing Zero Window acknowledgements in the OS TCP stack.

Have you ever tried "netstat -s" at receiver side?

I guess your system 10.113.130.171 of the new JumboTrace has problems with the Oflloading features (maybe it is like @cmaynard already mentioned a problem with the TCP checksum calculation during the offloading process). Otherwise I can´t explain packet #19.

@Luca, does it mean we're facing Kernel TCP bug, what do you think? Could you run the same test on later Kernel versions? Or the same test with artificially lowered WS factor?

@Christian_R what's that you don't like in JumboTrace packet 19? The sender sends 5 big 18k segments, every of them are split into 2 x 9k jumbo frames. One of these 9k-ers is lost on the path, the receiver signals about that via dup ACKs, the sender crafts 1 9k-er Fast retransmission, which happens to be only one had been lost. The receiver ACKs whole sequence of 5 18k segments. Am I missing something? For sure it is bad we don't use SACK.

Can the TCP Window Scaling factor be directly configured? I see it is now 2048 in the second registration for the failing connection (it used to be 4096 in the first), but not sure where they changed it

@Luca, as I've seen in a couple of articles , it can't be changed directly, but only by changing socket memory allocation settings.

@Packet_vlad: #19 is a partial retransmission of the 17946 byte large segment #11

And t does not happen only once in the trace, it does happen often; looks to bad for me.

@Christian_R, we don't have direct access to the machine for netstat -s, but we will ask for it.

@Luca: I missed it all the time. Please enable Selective ACK at the server side.

This is really really important.

Otherwise only normal ACK or Partial ACK algorithms will be used. The client uses it already:

For Redhat please use: echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf

Redhat Network Tuning

@Christian_R, will do that. I guess you actually meant tcp_sack = 1.

Of course mean 1. I have corrected it.

@Luca Have you solved the issue?

@Christian_R, the issue did not reproduced with Jumbo Frames and reduced TCP window, so that was enough to close the case. The root cause remains unknown.