ARP transmission delays promiscuous capture (libpcap/wireshark)

edit

Thanks for the reply. Noted on the timestamp algorithm. I don't know what wireshark does, but the libpcap app tries setting pcap_set_tstamp_type first as PCAP_TSTAMP_ADAPTER, then PCAP_TSTAMP_ADAPTER_UNSYNCED and finally PCAP_TSTAMP_HOST_HIPREC. If none of those are available, it just uses the default setting. On the test hardware, it ends up as default.

There is no chance the packets are delayed reaching the receiving machine. It is a direct point-to-point ethernet cable.

So either the packets are being timestamped incorrectly or there is a delay between the driver the kernel processing and the point libpcap applies the timestamp (before notifying user-space).

So why ARP?

I checked millions of packets. The indicated 999ms interpacket gap always happens just before the receiver sends the ICMP Port Unreachable message. Sure, correlation is not cause, but...

I don't know what wireshark does, but the libpcap app tries setting pcap_set_tstamp_type first as PCAP_TSTAMP_ADAPTER, then PCAP_TSTAMP_ADAPTER_UNSYNCED and finally PCAP_TSTAMP_HOST_HIPREC. If none of those are available, it just uses the default setting.

Wireshark doesn't try to set it at all. That means the time stamping is done by the host. It does request nanosecond precision if it's writing to a pcapng file, but that's it.

The indicated 999ms interpacket gap always happens just before the receiver sends the ICMP Port Unreachable message.

So why ARP? I don't see any mention of an ARP packet in what I quoted from your comment; I only saw a mention of an ICMP packet. Where are ARP packets involved?

That might be my misinterpretation. I've have both wireshark sessions (TX and RX) running before plugging in the cable. Plug in the cable and the sender sends an ARP to resolve the MAC address and RX responds with an ICMP/3 Port Unreachable in return. That handshake happens only at the very beginning.

After that RX periodically sends the ICMP Port Unreachable packet. I actually don't know why: either as a "follow up" to the initial ARP or because the inbound UDP packets, truly in fact, do not have a port open on the destination.

But if we are promiscuously sniffing, then there never will be a destination port open.

I don't know if this is related to the initial ARP or not. Just that the 999ms delay always happens just before the ICMP is sent out...

Thanks Guy, that's helpful. Each machine has two Ethernet ports. The two machines are directly connected by an ethernet cable. No switches or routers in between. netstat -vaun shows one udp port open on 192.168.16.195 (IP for eth1): 123 [NTP].

The sender sends to 192.168.16.195:5555. The ICMP/3 Port Unreachable says that 5555 is unreachable. That's true.

In the end, I don't know what the connection is here. Just that libpcap stops capturing for 999ms just before the kernel sends the port unreachable notice... Not sure where to look next.