There are some packets missing at the beginning of the large data-transfer. Every time the client receives a segment it did not expect yet, it sends a DUP-ACK. The server responds witha fast-retransmission after receiving the 3rd DUP-ACK (since the trace was made on the client side, it looks like it sends them out only after receiving more than 3 DUP-ACKs as packets need to travel back and forth).
This mechanism works well, until the client asks for the segment starting at sequence number 29567. This segment does not get retransmitted by the server (or gets lost on the way). But since there are no RTO retransmissions, it looks like the connection with the server is lost. The client stops sending DUP-ACKs by then as no more data comes in (the server is done sending data as you can see by the non-full-MSS segment in frame 189). It is up to the server to make sure the data gets to the client and is acknowledged.
After 7200 seconds the TCP stack sends out a TCP-Keep-Alive, which most OSses do by default. The server however does not recognize the TCP session anymore and sends a TCP-RST.
Can you tell us what kind of devices are in between the client and the server? I see a TTL of 61 for the server packets, which suggests 3 routing hops between the server and the client. Are there any firewalls and or loadbalancers involved? Or just routers/L3-switches? I'm particularly interested in these kind of devices as they are session based and might intervene.
As the problem only occurs sporadically, are you aware of using dumpcap instead of wireshark to do the capturing? You can use the -b options to create a ring-buffer to capture for a long time without filling your disk. Have a look at dumpcap in your wireshark folder and use "dumpcap -h" for options.
The server side trace would be the most interesting one in this case as from the clients perspective it stops transmitting data (which might or might not be the case).
We much prefer to analyze packets instead of pixels. Are you able to share the capture file on a public file share? Have a look at Tracewrangler if you need to anonymize the file.
If possible, a trace on both the client and server side would help the analysis.
Interesting little tool. Here's the anonymized packets in that stream. http://streaming2.thedavidcorrigan.co...
We're working on getting a capture from both sides. Right now we have the server owners, and the network team trying to figure this out and I've been bouncing ideas off both of them while we all try and figure it out. It's infrequent enough to be difficult to pin down but frequent enough to annoy a lot of people running big jobs.