Ask Your Question
0

ARP transmission delays promiscuous capture (libpcap/wireshark)

asked 2019-03-27 14:19:06 +0000

Danny gravatar image

updated 2019-03-28 00:59:18 +0000

I have two machines: TX and RX. RX runs a libpcap application to sniff out and receive traffic on a dedicated ethernet port.

TX transmits a continuous stream of UDP packets holding MPEG-TS.

Whether using the libpcap app or wireshark, every once in a while there is gap between received packets of almost 1 second.

I've tracked this down to something to do with ARP.

TX Machine

Running wireshark on the sender shows packets being sent at regular intervals: TX Wireshark Dump

RX Machine

Here's what wireshark shows on the receiver:

RX Wireshark Dump

There is a 999ms gap in receiving the two packets just before the ICMP packet sent by the receiver.

The time gap is very consistent and presents with both Wireshark and compiled libpcap app written in C++. Time gap always just before ICMP is sent by RX.

The receiver must be fully promiscuous and faithfully capture and report what's on the wire.

Disabling the ICMP port unreachable messages by setting arp_ignore is not a solution; the sender cannot send any data without knowing the MAC address of RX.

Is this a problem with libpcap?

This seems like purely a receiver-side issue but much searching has not come up with any solution.

I need a faithful capture from the wire. Something to modify somewhere?

Reference

The libpcap app is using the latest version, 1.9.0. Wireshark says:

Running on Linux 4.4.127-1.el6.elrepo.i686, with locale en_US.UTF-8, with libpcap version 1.9.0-PRE-GIT (with TPACKET_V3), with libz 1.2.3, GnuTLS 2.8.5, Gcrypt 1.4.5, without AirPcap.

EDIT I

Note that the machines are directly connected with an Ethernet cable. There is no network, router, or switches in between.

edit retag flag offensive close merge delete

Comments

  1. You should modify your post to ask 1 specific question
  2. It sounds like you are experiencing a delay between two machines on a network, which does occasionally happen.
Ross Jacobs gravatar imageRoss Jacobs ( 2019-03-27 15:22:50 +0000 )edit

To truly determine what's happening on the wire you should also capture off machine, preferably with a tap.

grahamb gravatar imagegrahamb ( 2019-03-27 16:58:00 +0000 )edit

I've added a note to the question: the machines are directly connected with an Ethernet cable. There is no network, router, or switches in between.

Also, see my comment to Guy. The behavior is very consistent: the packet just before the ICMP goes out has delayed reception.

Danny gravatar imageDanny ( 2019-03-28 00:58:33 +0000 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-03-27 19:00:07 +0000

Guy Harris gravatar image

Is this a problem with libpcap?

Unlikely - on Linux (and most other platforms), libpcap doesn't time-stamp the packets, the OS networking stack or the OS capture mechanism, in kernel mode, time-stamps the packets.

Probably either 1) the packets really are getting delayed in getting to the receiver or 2) there's a delay between the arrival of the packet on the receiver's network adapter and a time stamp being applied to it by the Linux networking stack.

BTW, in what fashion is ARP involved here?

edit flag offensive delete link more

Comments

Thanks for the reply. Noted on the timestamp algorithm. I don't know what wireshark does, but the libpcap app tries setting pcap_set_tstamp_type first as PCAP_TSTAMP_ADAPTER, then PCAP_TSTAMP_ADAPTER_UNSYNCED and finally PCAP_TSTAMP_HOST_HIPREC. If none of those are available, it just uses the default setting. On the test hardware, it ends up as default.

There is no chance the packets are delayed reaching the receiving machine. It is a direct point-to-point ethernet cable.

So either the packets are being timestamped incorrectly or there is a delay between the driver the kernel processing and the point libpcap applies the timestamp (before notifying user-space).

So why ARP?

I checked millions of packets. The indicated 999ms interpacket gap always happens just before the receiver sends the ICMP Port Unreachable message. Sure, correlation is not cause, but...

Danny gravatar imageDanny ( 2019-03-28 00:38:49 +0000 )edit

I don't know what wireshark does, but the libpcap app tries setting pcap_set_tstamp_type first as PCAP_TSTAMP_ADAPTER, then PCAP_TSTAMP_ADAPTER_UNSYNCED and finally PCAP_TSTAMP_HOST_HIPREC. If none of those are available, it just uses the default setting.

Wireshark doesn't try to set it at all. That means the time stamping is done by the host. It does request nanosecond precision if it's writing to a pcapng file, but that's it.

The indicated 999ms interpacket gap always happens just before the receiver sends the ICMP Port Unreachable message.

So why ARP? I don't see any mention of an ARP packet in what I quoted from your comment; I only saw a mention of an ICMP packet. Where are ARP packets involved?

Guy Harris gravatar imageGuy Harris ( 2019-03-28 03:23:20 +0000 )edit

That might be my misinterpretation. I've have both wireshark sessions (TX and RX) running before plugging in the cable. Plug in the cable and the sender sends an ARP to resolve the MAC address and RX responds with an ICMP/3 Port Unreachable in return. That handshake happens only at the very beginning.

After that RX periodically sends the ICMP Port Unreachable packet. I actually don't know why: either as a "follow up" to the initial ARP or because the inbound UDP packets, truly in fact, do not have a port open on the destination.

But if we are promiscuously sniffing, then there never will be a destination port open.

I don't know if this is related to the initial ARP or not. Just that the 999ms delay always happens just before the ICMP is sent out...

Danny gravatar imageDanny ( 2019-03-28 03:33:33 +0000 )edit

After that RX periodically sends the ICMP Port Unreachable packet. I actually don't know why: either as a "follow up" to the initial ARP or because the inbound UDP packets, truly in fact, do not have a port open on the destination.

ICMP Port Unreachable messages are NOT sent as followups to an ARP packet; they're sent because the inbound UDP packets, truly in fact, do not have a port open on the destination.

If the machine sending the ICMP Port Unreachable is on the same LAN segment as the machine to which it's sending it, but doesn't know the MAC address of that machine, or if they're not on the same LAN segment and the machine sending the ICMP Port Unreachable doesn't know the MAC address of the router for the first hop of the route to that machine, it will have to ...(more)

Guy Harris gravatar imageGuy Harris ( 2019-03-28 04:17:42 +0000 )edit

Thanks Guy, that's helpful. Each machine has two Ethernet ports. The two machines are directly connected by an ethernet cable. No switches or routers in between. netstat -vaun shows one udp port open on 192.168.16.195 (IP for eth1): 123 [NTP].

The sender sends to 192.168.16.195:5555. The ICMP/3 Port Unreachable says that 5555 is unreachable. That's true.

In the end, I don't know what the connection is here. Just that libpcap stops capturing for 999ms just before the kernel sends the port unreachable notice... Not sure where to look next.

Danny gravatar imageDanny ( 2019-03-28 05:53:44 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2019-03-27 14:19:06 +0000

Seen: 493 times

Last updated: Mar 28 '19