 | 1 | initial version |
The most useful statistic for investigating throughput problems is bytes-in-flight (BIF) and perhaps no one has shown such a Wireshark chart because either the traffic must be captured at the sender’s end or its long-standing bug can present misleading information. I have used another free tool that complements Wireshark and avoids both weaknesses. My BIF charts have variously been overlaid with graphs of the receive window, a data-sequence graph to show the context of packet losses and other abnormalities, network transit times, and a graph of data throughput. They show how the various flow-control decisions and network events affect throughput. Unfortunately WS won’t allow me to show you the charts here but they can be found through the link in my profile.
For the ‘slow’ capture as seen by the sender, bytes-in-flight rose rapidly in slow-start mode until, with almost 8 Mbytes in flight, there were heavy packet losses. Initially, packets were lost at a regular rate of 27 in every 30, consistent with tail drops from a queue formed where link speed dropped by a factor of ten. Two thirds of BIF (5.3 MB) was lost at this stage.
The sender responded quickly to the selective acks from the receiver, but after a large burst of retransmissions the sender apparently ignored some of the later SACKs and became particularly cautious, retransmitting only one packet and roughly doubling the number of packets in successive round-trips as in a slow-start mode. There were two such sequences of doubling retransmissions starting with one packet. Consequently, recovery of all the lost data took nearly six seconds. Then the congestion window was set very small and opened slowly in congestion avoidance mode. These flow-control events were responsible for the overall slow throughput.
Throughput climbed slowly to only 90 Mbps and there were no more packet losses. Unfortunately, file transfers that begin with an aggressive slow-start build-up and produce heavy packet losses like this are very common. I agree with the consensus that buffer bloat from this transfer traffic alone should be avoided by limiting the receive window to 7 MB or less.
The BIF chart for the fast transfer tells a quite different story. This TCP driver had set a slow-start threshold at only 2 Mbytes and that avoided packet losses entirely. Near the end, when BIF reached 5.2 MB and throughput was about 210 Mbps, some packets must have arrived out of order. The resulting SACKs prompted a few retransmissions and corresponding reductions in the congestion window, in three steps of roughly 30% each. We know the retransmissions were unnecessary because each one generated a D-SACK from the receiver. Without more samples we can’t tell why a slow-start threshold was set at such a low point, but I believe many drivers will do this after recovering from large packet losses.
A more important question is why, with the path’s bottleneck speed said to be 1 Gbps, packets were overtaken when BIF was about 5 MB and throughput was below 210 Mbps? The most likely cause is heavy foreign traffic sharing critical parts of the network, and one direction might always be faster because either the sender has a TCP driver that sets a useful slow-start threshold, or the competing, unseen traffic flows mostly in one direction. The BIF patterns from the two simultaneous captures are so different from the slow and fast captures we wonder if they came from a different network. The saw-tooth shape of the BIF graph never exceeded 3 Mbytes and throughput never reached 10 Mbps.
With captures taken at both ends we can plot the transit times of all the packets flowing from the sender to the receiver, and also see that many packets arrived out of order. There were many short periods without overtaken packets, and they occurred only when BIF and throughput were very low. Most of the variations in transit times ranged within one millisecond but for very brief periods, at intervals of roughly 800 ms, longer queues must have formed somewhere in the network, extending transit times by an extra millisecond. The longer queues always coincided with sudden drops in BIF. As Phil Storey has noted, no data packets were lost but the many overtaken packets generated SACKs that occasionally prompted retransmissions with consequent reductions in the congestion window. Those congestion-window closures crippled throughput.
The arrival of packets out of order implies use of an another path and breaks a cardinal rule of TCP good practice. Perhaps some router or firewall sent packets along a different path because there was no space in the queue for the main path.
This wide variation in network behaviour shows how performance can be affected by so many uncontrolled variables and reinforces the importance of looking carefully at a problem. Were the sequence gaps the result of packet losses in the network or the sniffer? Were the retransmissions caused by packet losses or packets out of order? In this case more traffic samples are needed, particularly to learn the rules governing the use of slow-start thresholds. We must look at the whole picture as well as the details, and the world’s best packet microscope is not always the best visibility tool.
