slow transfer in one direction

edit

[Edit] Note: This answer applies to the newer "dest" and "src" PCAPs. The behaviour there is very, very different than in the original "slow" PCAP - where there are very real packet losses and retransmissions. Thus, this answer shouldn't be compared to the other answers here.

The "slow" PCAP deserves its own separate answer.

In "src", I can see that the slow throughput is due to the sender going into congestion avoidance mode due to apparent, but not real, packet losses. Severe out-of-order events (such as one full-sized packet overtaking 9 other full-sized packets) are happening regularly.

For such an example, have a look at packet #53529 in the "src" PCAP. Observe the next 9 data packets as well as the intervening SACKs.

When a packet arrives ahead of where it should be, the receiver sends SACKs implying that packets were missing. However, the missing packets arrive very quickly afterwards (sub millisecond).

When the OOO events are severe enough (very often) the SACKs make the sender believe that there were real packet losses and so it halves its transmit window and then ramps up slowly. Sometimes the sender actually retransmits data packets - causing the receiver to send D-SACKs (indicating that data was received twice).

When 3 transmit window "halvings" occur close together, we end up reducing the transmit window by a factor of 8. The common "increase the transmit window by just one extra packet per round trip" mechanism means that throughput is dramatically reduced.

So your problem is packets becoming OOO. Where and why would full-sized packets overtake several other full-sized packets in your network? I stress the "full-sized" because it is more common for very small packets to overtake big ones.

I'll add some more packet examples when I've had more time to look at the PCAPs.

This looks like a buffer bloat issue. Data is sent on a high speed network and needs to be forwarded onto a WAN connection with lower bandwidth, this results in buffering. But when the buffer fills up, packets will be discarded. Once the packet-loss is detected, the sending side will reduce it's congestion window and this means less data can be sent per roud-trip time. This results in reduced bandwidth.

In the slow transfer, the max window size seen is ~16MB, this is probably larger than the buffer on the WAN router, which is the why the buffer bloat occurs.

In the fast transfer, the max window size seen is ~256KB, perhaps this was preventing the sending system to overload the buffer.

It could also be that there is different equipment with different buffer sizes on both locations, causing different behavior.

You could try to limit the maximum window size on the receiving system of the slow response. See if it makes a difference.

Most remarkably, the slow transfer file shows quite a number of retransmissions, where the fast transfer does not show so many. I have plotted the number of packets send by the server vs. the retransmissions (in red).

It's interesting to see, that the time interval of 44 sec onwards has ACK-times of up to 2 seconds. Try the display filter tcp.analysis.ack_rtt > 1 for fun.

The server has a constant TTL of 124. Is it possible, that a topology change has happened in the WAN or in a carrier network (SD-WAN or whatever)? Or a failover on L2 somewhere? A failover from one WAN-link to another?

@SYN-bit could buffer bloat cause these looong RTTs?