How does wireshark determine if a TCP packet is out-of-order?

edit

Some comments in the source code of the TCP dissector kind of explain it:

/* RETRANSMISSION/FAST RETRANSMISSION/OUT-OF-ORDER
 * If the segment contains data (or is a SYN or a FIN) and
 * if it does not advance the sequence number, it must be one
 * of these three.
 * Only test for this if we know what the seq number should be
 * (tcpd->fwd->nextseq)
 *
 * Note that a simple KeepAlive is not a retransmission
 */

So, first condition to match is that the received segment does not have a sequence number that is equal to (or higher than) the next expected sequence number.

Then later on (after checking if the packet was a fast-retransmission):

    /* If the segment came relatively close since the segment with the highest
     * seen sequence number and it doesn't look like a retransmission
     * then it is an OUT-OF-ORDER segment.
     */

And in the code below that, it compares the delta time between this frame and the time of the frame with the highest received sequence number. If that delta is less than the initial round-trip time (or less than 3 ms when the 3-way-handshake was not captured), it is considered an out-of-order packet.

So in the case above, the proper order for packets from X to arrive would have been:

11. 198.41.209.139 → 192.168.1.100: SEQ=8205 – ACK=661 [ACK]
14. 198.41.209.139 → 192.168.1.100: SEQ=9645 – ACK=661 [ACK]
15. 198.41.209.139 → 192.168.1.100: SEQ=11085 – ACK=661 [ACK]
12. 198.41.209.139 → 192.168.1.100: SEQ=12525 – ACK=661 [PSH, ACK]

You could say that only frame 12 was received out-of-order, the rest was fine.

However, Wireshark interprets the packets as they come in and only keeps little state of the session to base the analysis on. This means when the packets arrive in this order:

11. 198.41.209.139 → 192.168.1.100: SEQ=8205 – ACK=661 [ACK]
12. 198.41.209.139 → 192.168.1.100: SEQ=12525 – ACK=661 [PSH, ACK]
14. 198.41.209.139 → 192.168.1.100: SEQ=9645 – ACK=661 [ACK]
15. 198.41.209.139 → 192.168.1.100: SEQ=11085 – ACK=661 [ACK]

The only thing it knows when packet 12 arrives is that there is data missing, hence the "Previous Segment not captured". Then when it receives frame 14, the sequence number is not advancing (as frame 12 has a higher sequence number) and the frame was received quickly, so it could not have been a retransmission. Therefor it labels it as an out-of-order. The same goes for packet 15.

Doing this from a screenshot is suboptimal at best, especially without packet numbers, but let's try. If I get it wrong I'm sure @SYN-bit and @Packet_vlad will correct me :-)

One thing is easy: packet #12 has a higher sequence number than packets #14 and #15, so there's a gap before it and that leads to the symptom "previous segment not captured".

This also makes both #14 and #15 either a retransmission or out-of-order, because they both should have arrived before #12 (and still #14 before #15). Wireshark has some logic to determine if a packet arriving late is a retransmission or just switched places during travel across the network. Basically it tries to determine if the sender could have known that a packet was lost or not, and that's depending on the initial round trip time (iRTT). We don't know the RTT here because it's just a screenshot, so it's either that 14 and 15 arrived faster than iRTT, or arrived within a couple of milliseconds after where they should have arrived.

So no, 1st concern assumption is not correct. #14 coming before #13 is not the problem. Both #14 and #15 should have arrived before #12.

2nd concern: the symptom in #12 means both #14 and #15, not just #15. Wireshark sees that a sequence number in #12 is higher than what's expected, so it states "previous segment not captured", which may be a bit misleading in this case - it should be "previous segment(s) not captured")