Revision history [back]

Your throughput limitation isn’t related to Receive Window sizes (or network bandwidth limits). It is entirely limited by the rate that the sending server is delivering data to the network. It would appear to be a send buffer limitation (but could be FTP delivering data to TCP in the server) with a buffer size of 128 KB.

There are never more that 128 KB of bytes in flight and data is sent in blocks of 64 KB. So there are never more than 2 x 64 KB “blocks” in flight. A new 64 KB “block” cannot begin to be transmitted until the whole 64 KB (from two bursts ago) is acknowledged.

Unfortunately, due to capturing inside the server, we are seeing packets that appear to be very large – but would be broken up into normal sized packets after we captured them – and as they leave the server.

The bursts of 64 KB always appear as one very large 63,648 byte packet followed by a smaller 1888 byte packet. The agreed MSS is 1260 and so in real life, our 64 KB burst would appear on the wire as 52 or 53 normal sized packets. The last packet in the burst has a PUSH flag. The sender has to wait for the ACK to that last packet before it starts a new 64 KB block.

You need to check your server configuration for anything that would limit the output to 128 KB per RTT. Possible culprits could be send buffer space limits in TCP, a setting in the FTP server or something else.

At the network or client side, you should investigate the “extra 45 ms” that will make more sense when you see the charts below.

It would be good to see a simultaneous capture from both the sender and receiver.

Some TCP-Trace charts tell the story.

The flow rate (slope of the packets) is very consistent for most of the flow, but we do see an increase just before the end. Something changes in the blue circle.

In the consistent part of the early flow, there are 2 x 64 KB bursts every 125 ms. The minimum RTT is 76 ms and we see the ACK for the first packet in the burst arrive after 76 ms. The ACK for the second-last packet in the burst arrives after 80 ms.

We can deduce that if we had a packet capture taken at the receiver, the stack of real-life normal sized packets would look similar to the ACK line in the blue circled area. That is, the last packet would be 4 ms behind the first.

The most important piece of information on this chart is that the ACK for the very last packet in the 64 KB burst is always an extra 45 ms later than we might expect.

The red circles show the ACK line step up 125 ms after the original packet was seen. The usual 80 ms plus the extra 45 ms gives us the total 125 ms time per 2 x 64 KB.

Could the additional 45 ms be a “Delayed ACK” if there were an odd number of real packets in the 64 KB? This would mean that the receiver has an unusually short delayed ACK timer.

Could the last packet in the burst always get delayed somewhere in the network? Or the ACK delayed on the way back?

We can’t get definitive answers just from this capture – but at least we can see what is happening (and consistently). Our throughput is 128 KB per 125 ms, which would equate to 1 MB/s or ~8 Mbps.

At time 55.5 seconds, the extra 45 ms goes away and we enter a state where we get the 2 x 64 KB bursts being transmitted together. This effectively increases the throughput to 128 KB per 80 ms (1.6 MB/s or ~12.8 Mbps).

Your throughput limitation isn’t related to Receive Window sizes (or network bandwidth limits). It is entirely limited by the rate that the sending server is delivering data to the network. It would appear to be a send buffer limitation (but could be FTP delivering data to TCP in the server) with a buffer size of 128 KB.

There are never more that 128 KB of bytes in flight and data is sent in blocks of 64 KB. So there are never more than 2 x 64 KB “blocks” in flight. A new 64 KB “block” cannot begin to be transmitted until the whole 64 KB (from two bursts ago) is acknowledged.

Unfortunately, due to capturing inside the server, we are seeing packets that appear to be very large – but would be broken up into normal sized packets after we captured them – and as they leave the server.

The bursts of 64 KB always appear as one very large 63,648 byte packet followed by a smaller 1888 byte packet. The agreed MSS is 1260 and so in real life, our 64 KB burst would appear on the wire as 52 or 53 normal sized packets. The last packet in the burst has a PUSH flag. The sender has to wait for the ACK to that last packet before it starts a new 64 KB block.

You need to check your server configuration for anything that would limit the output to 128 KB per RTT. Possible culprits could be send buffer space limits in TCP, a setting in the FTP server or something else.

At the network or client side, you should investigate the “extra 45 ms” that will make more sense when you see the charts below.

It would be good to see a simultaneous capture from both the sender and receiver.

Some TCP-Trace charts tell the story.

The flow rate (slope of the packets) is very consistent for most of the flow, but we do see an increase just before the end. Something changes in the blue circle.

In the consistent part of the early flow, there are 2 x 64 KB bursts every 125 ms. The minimum RTT is 76 ms and we see the ACK for the first packet in the burst arrive after 76 ms. The ACK for the second-last packet in the burst arrives after 80 ms.

We can deduce that if we had a packet capture taken at the receiver, the stack of real-life normal sized packets would look similar to the ACK line in the blue circled area. That is, the last packet would be 4 ms behind the first.first. Based on the rate we see ACKs, it would appear that there is an available bandwidth of somewhere around 100 Mbps. 

The most important piece of information on this chart is that the ACK for the very last packet in the 64 KB burst is always an extra 45 ms later than we might expect.

The red circles show the ACK line step up 125 ms after the original packet was seen. The usual 80 ms plus the extra 45 ms gives us the total 125 ms time per 2 x 64 KB.

Could the additional 45 ms be a “Delayed ACK” if there were an odd number of real packets in the 64 KB? This would mean that the receiver has an unusually short delayed ACK timer.

Could the last packet in the burst always get delayed somewhere in the network? Or the ACK delayed on the way back?

We can’t get definitive answers just from this capture – but at least we can see what is happening (and consistently). Our throughput is 128 KB per 125 ms, which would equate to 1 MB/s or ~8 Mbps.

At time 55.5 seconds, the extra 45 ms goes away and we enter a state where we get the 2 x 64 KB bursts being transmitted together. This effectively increases the throughput to 128 KB per 80 ms (1.6 MB/s or ~12.8 Mbps).

If you can get the sender to transmit more than 128 KB per round trip, you should be able to get closer to the expected 100 Mbps available.