Wireshark sees new UDP packet, but application receives OLD UDP packet
We have an application that receives multicast packets from a device over a 10Gb direct attach link. The 15KB UDP packets are coming in at 50Mbits/s, which is well within the data rate that the receiving application can handle. The packets contain sequence numbers. Strangely, the receiving application often receives a packet with an OLD sequence number that takes the place of the expected packet, always somewhere in the middle of the receive sequence. When we look at the incoming packets with Wireshark, however, Wireshark sees the expected packet sequence.
The data traffic consists of multiple multicast streams with synchronized packet sequence numbers. When the error occurs, it can occur on any combination of the streams, not always all of them, but the affected packet number AND the old packet number is the same for all streams that have this error.
With a different configuration: 49KB packets coming in at 600Mbits/s, we do not see any errors. Also, when we test the multicast transmission at 50Mbit/s using a PC to generate the packets, we do not see any errors.
What could cause the correct packet seen by Wireshark to be replaced by an old packet when it gets to the application?
Where do you do the capture with Wireshark? On the same system where the receiving application is running, or on a dedicated capture system?
On the same system as the receiving application.
How do sequence number and old sequence number correlate? n to n-1, or something else?