Increasing robustness of a TCP receiver against peaks of CPU usage

asked 2018-07-17 20:58:13 +0000

EricC gravatar image

Using wireshark to diagnose and solve data loss, data corruption and eventual socket connection loss. These happen when my receiver of a TCP socket is not fast enough to read the incoming packets from a remote data acquisition board.

The setup:

Our data receiver code runs on a PC with Windows 7 and can also run on an embedded. Same problem is seen on both platforms.

A remote data acquisition board sends samples at 240KB/second over tcp.

When the priority of the data receiving thread is low and CPU usage peaks (e.g. because I run Wireshark at high priority on the same CPU core as the data receiving code), we see data loss, data corruption and connection loss.

The question:

I expected the Window value on the ACK messages to decrease towards 0, and only then start having trouble. Then I would conclude that I need to somehow increase the the size of the receiving buffer to make my receiver more robust against peaks of CPU usage.

But the win value does not seem to even approach zero at the time that we start seeing Dup ACKs, which coincides with data corruption.

Shouldn't we see the Win value on ACK messages decrease to zero as the receiving buffer overflows (since the receiver is too slow to read the data as fast as needed)?

Then an "easy" fix should be to increase the receiving buffer (to minimize issues when a peak of CPU usage makes the receiver transiently slow).

edit retag flag offensive close merge delete

Comments

BTW, we are using an ASIX Ethernet to USB bridge. So Wireshark is actually talking via USB to the ASIX bridge.

EricC gravatar imageEricC ( 2018-07-17 21:01:34 +0000 )edit

Wireshark never talks, it listens (and captures). What side of this bridge is connected where? USB on the PC, Ethernet on the acquisition board? What if you hook a tap in between the bridge and the acquisition board?

Jaap gravatar imageJaap ( 2018-07-18 04:37:34 +0000 )edit

Jaap, what I meant was that Wireshark (I suppose) needs to interact with the Network driver to grab TCPIP data. But since we are using a USB2Ethernet ASIX chip, everything happens over USB. I thought this could be relevant in terms of performance and possibly more than that.

The USB side of the bridge is connected to the PC. Ethernet side connected to the acq board.

When I force the receiving code on the PC to be slow (lowering priority and having other apps at high priority), the log shows that ACKs are no longer sent from the PC side after each 3 data packets. Instead I see 40 or 50 data packets coming from the acq board without ACK from the PC. Then Wireshark complains about a missing packet and trouble starts. But the win value is always close to 65K. I am puzzled.

EricC gravatar imageEricC ( 2018-07-18 11:16:32 +0000 )edit