ESP Packets Expected SN
I have several pcaps I pulled from a Cisco Switch using the web interface, which is a really convenient way to grab a capture of a switch port, no acl, no span, easy peasy, it's the Embedded Packet Capture feature. I preface my question because initially I was going to post this about some Expert Warnings I'm seeing. I'm capturing from both ends of an IPSec tunnel that has no known issue, I wanted to see what I could see so when I do have an issue I had a benchmark to refer to.
But come to find out about 20% of my captured packets have expected sequence number errors, and there are many SN missing in each ESP packet.
However, if you sort the Expert Info dialog box by protocol and expand ESP, every single time I grab one of these captures, which I've done about a dozen times now, the expected SN warnings begin with packet number 502. These captures from the web UI cap out at 100Mb, so it's upwards of 70k packets each time. But, if i limit the capture to 500 packets instead, as expected no errors.
Version I'm using to view these on our datacenter mgmt servers is 4.2.6 and 4.2.9 so a little old, I haven't tried it on the latest 4.4.2 yet.
Clearly I'm reaching some sort of critical mass, has anyone ever heard of an issue like this on the Wireshark side?
About the only other thing that's different is pulling the cap from the interface directly, through the Cisco Web UI, but I'm having trouble working that out in my head that it's problematic to do so...
Just to add a little color, no matter when the capture is taken, the expected sequence number errors start each time at packet 502 and it's always 184 sn's off....so if packet 501 is 20097935, then packet 502 will be 20098119
So you are saying that when you do a capture, and no matter where the sequence number is, you get ~500 frames with no SNs missing, then a drop of ~184 frames, then more drops?
Have you checked the time deltas between the first 500 frames and the gap(s) that come later? Were the IP identifiers coming out sequentially during the good 500 frame burst? If you are able to decrypt (if not null) the payload, can you detect that data has been missed?
This sounds like a characteristic/limitation of the capture mechanism you are using, (which I am not familiar with), rather than a problem with Wireshark.
No matter when I take a capture the first 501 packets have no issue. Literally packet number 502 is an issue. However there’s no known issue on the link or interface or the IPsec tunnel.
I don’t actually think it’s a Wireshark issue, I’d agree it’s something with the capture method. I was hoping someone has just seen this behavior before.
Thanks for the input.
Having multiple ESP streams in packet captures can be quite perplexing. In Expert mode, the SPI is displayed. Is this the same SPI as in the first 501 packets?
Try this
Create a column by SPI and then sort the column. Filter by ESP stream. esp.spi == x. x is the SPI in the packet.
Won’t be back to that site for two days. I’ll update accordingly after..
Yup, same SPI thorugh the whole capture...
I apologize for missing your post. I'm surprised the ESP tunnel is still working. The anti-replay window is either very large or disabled. In my experience, the IP ID updates sequentially. Could you add a column for IP ID and verify if the IP ID increased by 184?
The sequence number is off, in the ESP header, not the IP ID... <edit> Clearly I was not on the same page. I just double checked and yes the IP ID also jumps up by 184 as well. It's interesting because the IP ID is just incrementing by one the whole time, but yes it bumps up following the ESP sequence numbers, and after packet 501 each time there's a bump in the SN's there's a bump in the ID's, but no issue on the link. It's as if there's packet loss after 501 packets.
Great!! Can you do a packet capture at the source of the IPSEC tunnel? If the missing IP IDs and sequence numbers appear in the IPSEC source capture, it indicates that your other capture point might not be capturing all the data, or there could be a network issue. To check if the drops are real, try making the anti-replay window smaller than the 184-byte gap. If the gap is legitimate, the IPSEC tunnel will drop. You can also capture packets at the IPSEC tunnel's destination. Remember to restore the anti-replay window back to its original settings to avoid complaints.
That may take some time. I don’t have access. That’s another group. But as I mentioned there’s no known issue with the tunnel, I just happened to notice this anomaly with the sequence numbers. And the tunnel is created from hardware encryption devices, not a firewall, just to be clear. I had just pulled the pcap from a Cisco switch in the middle of the path that I manage.
The IP Identification (IP ID) field, as per the RFCs, doesn't need to be unique and primarily serves IP fragmentation. The majority of vendors increment the IP ID with each subsequent packet they send. I use it to identify missing packets in an encrypted stream.