Capture Filter not working due to incorrect BPF?

edit

If you're looking for an alternate work-around solution that is nearly identical to the correctly-generated BPF instructions from the original capture filter expression of (ether[len - 4:4] == 0x1d10c0da) and not (icmp or (vlan and icmp)), then you might try this filter:

(ether[len - 4:4] = 0x1d10c0da) and not (icmp or ((ether[12:2] = 0x8100 or ether[12:2] = 0x88a8 or ether[12:2] = 0x9100) and (ether[16:2] = 0x0800 and ether[27:1] = 1)))

This basically handles vlan the old-fashioned hard way, but a way that works regardless of the libpcap version. The generated BPF is as follows, which adds only 1 extra instruction to reload the Ethertype (i.e., the 2-byte value at ether[12:2]), since there's an implicit loading of that value when comparing the first icmp expression, as it first ensures that the Ethertype is IP before looking at the value of the IP header's Protocol field.

(000) ld       #pktlen
(001) sub      #4
(002) tax
(003) ld       [x + 0]
(004) jeq      #0x1d10c0da      jt 5    jf 18
(005) ldh      [12]
(006) jeq      #0x800           jt 7    jf 9
(007) ldb      [23]
(008) jeq      #0x1             jt 18   jf 9
(009) ldh      [12]
(010) jeq      #0x8100          jt 13   jf 11
(011) jeq      #0x88a8          jt 13   jf 12
(012) jeq      #0x9100          jt 13   jf 17
(013) ldh      [16]
(014) jeq      #0x800           jt 15   jf 17
(015) ldb      [27]
(016) jeq      #0x1             jt 18   jf 17
(017) ret      #262144
(018) ret      #0

Now if you're looking for an alternate work-around solution that produces identical correctly-generated BPF instructions, then you could try this filter:

(ether[len - 4:4] = 0x1d10c0da) and not ((ether[12:2] = 0x0800 and ether[23:1] = 1) or ((ether[12:2] = 0x8100 or ether[12:2] = 0x88a8 or ether[12:2] = 0x9100) and (ether[16:2] = 0x0800 and ether[27:1] = 1)))

This one explicitly loads the Ethertype to ensure it is IP before comparing the IP Protocol to ICMP, but in doing so allows the optimizer to only load that value once. Now we get the exact same output as bpfexam produces and which is 1 instruction fewer than the previous attempt:

(000) ld       #pktlen
(001) sub      #4
(002) tax
(003) ld       [x + 0]
(004) jeq      #0x1d10c0da      jt 5    jf 17
(005) ldh      [12]
(006) jeq      #0x800           jt 7    jf 9
(007) ldb      [23]
(008) jeq      #0x1             jt 17   jf 16
(009) jeq      #0x8100          jt 12   jf 10
(010) jeq      #0x88a8          jt 12   jf 11
(011) jeq      #0x9100          jt 12   jf 16
(012) ldh      [16]
(013) jeq      #0x800           jt 14   jf 16
(014) ldb      [27]
(015) jeq      #0x1             jt 17   jf 16
(016) ret      #262144
(017) ret      #0

the-tcpdump-group issue for the libpcap compiler output:
Capture Filter not working due to incorrect BPF? #1162

Yes, but in this case you get two issues for the price of one. :-)
I think there is a problem with the bpf generated by libpcap (at least 1.10.1).
And maybe wireshark should open the interface so that compiling the capture filter is more accurate.

I was hoping to test outside of Wireshark and dumpcap using the libpcap test program - filtertest - but was unable to recreate the dumpcap output.

filtertest and wireshark call pcap_open_dead() before compiling the capture filter.
filtertest.c:

pd = pcap_open_dead(dlt, snaplen);
if (pd == NULL)
    error("Can't open fake pcap_t");

if (pcap_compile(pd, &fcode, cmdbuf, Oflag, netmask) < 0)
    error("%s", pcap_geterr(pd));

wireshark - compiled_filter_output.cpp:

            pcap_t *pd = pcap_open_dead(device->active_dlt, WTAP_MAX_PACKET_SIZE_STANDARD);
            if (pd == NULL)
                break;
            g_mutex_lock(pcap_compile_mtx);
            if (pcap_compile(pd, &fcode, compile_filter_.toUtf8().data(), 1, 0) < 0) {
                compile_results.insert(interfaces, QString(pcap_geterr(pd)));

dumpcap opens the interface before compiling the capture filter. dumpcap.c:

    pcap_h = open_capture_device(capture_opts, interface_opts,
        CAP_READ_TIMEOUT, &open_status, &open_status_str);
...
    if (!compile_capture_filter(interface_opts->name, pcap_h, &fcode,
                                interface_opts->cfilter)) {
...
compile_capture_filter(const char *iface, pcap_t *pcap_h,
                   struct bpf_program *fcode, const char *cfilter)
{
...
    if (pcap_compile(pcap_h, fcode, (char *)cfilter, 1, netmask) < 0)

admin1@ubuntu2204:/usr/bin$ dumpcap --log-level debug -i 1 -f vlan -d --log-file /tmp/foo.out
Capturing on 'ens160'
(000) ldb      [vlanp]
(001) jeq      #0x1             jt 6    jf 2
(002) ldh      [12]
(003) jeq      #0x8100          jt 6    jf 4
(004) jeq      #0x88a8          jt 6    jf 5
(005) jeq      #0x9100          jt 6    jf 7
(006) ret      #262144
(007) ret      #0
admin1@ubuntu2204:/usr/bin$ cat /tmp/foo.out
 ** (dumpcap:1905481) 10:39:29.522753 [Capchild DEBUG] ./capture/capture-pcap-util.c:1578 -- open_capture_device(): Entering open_capture_device().
 ** (dumpcap:1905481) 10:39:29.522913 [Capchild DEBUG] ./capture/capture-pcap-util.c:1278 -- open_capture_device_pcap_create(): Calling pcap_create() using ens160.
 ** (dumpcap:1905481) 10:39:29.523013 [Capchild DEBUG] ./capture/capture-pcap-util.c:1280 -- open_capture_device_pcap_create(): pcap_create() returned 0x562799ce8770.
 ** (dumpcap:1905481) 10:39:29.523057 [Capchild DEBUG] ./capture/capture-pcap-util.c:1290 -- open_capture_device_pcap_create(): Calling pcap_set_promisc() with promisc_mode 1.
 ** (dumpcap:1905481) 10:39:29.523096 [Capchild DEBUG] ./capture/capture-pcap-util.c:1336 -- open_capture_device_pcap_create(): buffersize 2.
 ** (dumpcap:1905481) 10:39:29.523134 [Capchild DEBUG] ./capture/capture-pcap-util.c:1340 -- open_capture_device_pcap_create(): monitor_mode 0.
 ** (dumpcap:1905481) 10:39:29.541767 [Capchild DEBUG] ./capture/capture-pcap-util.c:1344 -- open_capture_device_pcap_create(): pcap_activate() returned 0.
 ** (dumpcap:1905481) 10:39:29.541841 [Capchild DEBUG] ./capture/capture-pcap-util.c:1649 -- open_capture_device(): open_capture_device SUCCESS : ens160

The linux code for libpcap does a check for vlan when opening the socket and sets a flag for the bpf code generator:
pcap-linux.c:

#if defined(SO_BPF_EXTENSIONS) && defined(SKF_AD_VLAN_TAG_PRESENT)
    /*
     * Can we generate special code for VLAN checks?
     * (XXX - what if we need the special code but it's not supported
     * by the OS?  Is that possible?)
     */
    if (getsockopt(sock_fd, SOL_SOCKET, SO_BPF_EXTENSIONS,
        &bpf_extensions, &len) == 0) {
        if (bpf_extensions >= SKF_AD_VLAN_TAG_PRESENT) {
            /*
             * Yes, we can.  Request that we do so.
             */
            handle->bpf_codegen_flags |= BPF_SPECIAL_VLAN_HANDLING;
        }
    }
#endif /* defined(SO_BPF_EXTENSIONS) && defined(SKF_AD_VLAN_TAG_PRESENT) */

gencode.c:

             * Do we need special VLAN handling?
             */
            if (cstate->bpf_pcap->bpf_codegen_flags & BPF_SPECIAL_VLAN_HANDLING)
                b0 = gen_vlan_bpf_extensions(cstate, vlan_num,
                    has_vlan_tag);
            else
                b0 = gen_vlan_no_bpf_extensions(cstate,
                    vlan_num, has_vlan_tag);