You would have to look into the capabilities of pcap-filter to know what you can filter on. Even though it does handle Geneve packets, VXLAN handling is not in there. Therefore it can't find the protocols you try to filter on on the locations in the packet where it expects it. Filter keywords, like geneve [vni] would adjust for that. Unfortunately for VXLAN it can't.