This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

tcpdump vs wireshark differences (packets merged?)

0

This started while trying figure out why loading a web page in a browser from apache via HTTP was so very slow going through a pinhole from an untrusted network to a trusted one.

I thought it was smoothwall related, see my post here http://community.smoothwall.org/forum/viewtopic.php?p=297864#p297864 but not so sure and drove me to using wireshark and tcpdump.

This is what I am seeing and lead to my main question of why does wireshark show 3 packets but tcpdump only shows 1 big packet? Is something combining them? Why do the ID's go in order for both wireshark and tcpdump (39771, 39772, 39773) and then wireshark goes to 700x but tcpdump goes to 39774? ug, out of my league here.

Wireshark on the wire shows 3 packets, id 7006, 7007, 7008 with a Total Length of 1500, however, on the router running tcpdump, I see ONE packet, id 39774 with a size of 4420 which causes the router to generate an ICMP Destination unreachable (Fragmentation needed).

    HTTP Request
    10:52:03.872268 IP (tos 0x0, ttl 127, id 58378, offset 0, flags [DF], proto TCP (6), length 795) 192.168.5.199.24914 > 10.0.0.16.http: Flags [P.], ack 1, win 64240, length 755
ACK
10:52:03.873744 IP (tos 0x0, ttl 64, id 39771, offset 0, flags [DF], proto TCP (6), length 40) 10.0.0.16.http > 192.168.5.199.24914: Flags [.], cksum 0x6c55 (correct), ack 756, win 6795, length 0

Start of HTTP response
10:52:04.075967 IP (tos 0x0, ttl 64, id 39772, offset 0, flags [DF], proto TCP (6), length 57) 10.0.0.16.http > 192.168.5.199.24914: Flags [P.], ack 756, win 6795, length 17
10:52:04.077207 IP (tos 0x0, ttl 64, id 39773, offset 0, flags [DF], proto TCP (6), length 1500) 10.0.0.16.http > 192.168.5.199.24914: Flags [.], ack 756, win 6795, length 1460

ACK
10:52:04.077315 IP (tos 0x0, ttl 127, id 58388, offset 0, flags [DF], proto TCP (6), length 40) 192.168.5.199.24914 > 10.0.0.16.http: Flags [.], cksum 0x862a (correct), ack 1478, win 64240, length 0

ok, this i don't understand. tcpdump on the router says it gets one packet, id: 39774 with a size of 4420! yes, that is bigger than MTU, so the following ICMP would be correct. however, wireshark running on the wire saw 3 packets, IDs (7006, 7007, 7008) each with size of 1500, but saw no packet with id of 39774. tcpdump never showed the 7006-8 packets.` 10:52:04.077647 IP (tos 0x0, ttl 64, id 39774, offset 0, flags [DF], proto TCP (6), length 4420) 10.0.0.16.http > 192.168.5.199.24914: Flags [.], ack 756, win 6795, length 4380

10:52:04.077679 IP (tos 0xc0, ttl 64, id 57158, offset 0, flags [none], proto ICMP (1), length 576) gate > 10.0.0.16: ICMP 192.168.5.199 unreachable - need to frag (mtu 1500), length 556
        IP (tos 0x0, ttl 64, id 39774, offset 0, flags [DF], proto TCP (6), length 4420) 10.0.0.16.http &gt; 192.168.5.199.24914:  tcp 4384 [bad hdr length 16 - too short, &lt; 20][|icmp]</code></pre><p>Edit 8-may-2011:</p><p>Network "diagram":</p><p>192.168.5.199 client -&gt; vSwitch-X -&gt; Smoothwall -&gt; vSwitch-Y -&gt; 10.0.0.16 server</p><p>WireShark is on a separate machine on vSwitch-Y</p><p>tcpdump is on the server interface of the smoothwall (vSwitch-Y facing)</p><p>I'm now using netcat to transfer/receive 1MB of text data (a-z repeated many times).</p><pre><code>nc {IP} 9876 &lt; data-in        # sender

nc -l -p 9876 > data-out # receiver

If Debian is the source machine, very many ICMP Fragmentation Required packets are generated by the router.

If Windows XP is the source machine, transfer is very fast and no ICMPs.

vmware-tools and vmxnet3 driver is installed on everything (debian, windows, router)

Edit: 9-may-2011

While the issue was not related to wireshark or tcpdump, it was the strange packets I was seeing that sent me here to ask “why” which may have got me looking in the right place.

asked 05 May ‘11, 08:53

gregg's gravatar image

gregg
1114
accept rate: 100%

edited 08 May ‘11, 09:17

i forgot to mention that everything is inside an ESXi 4.0 server running on a dell poweredge 2900 with a broadcom netxtreme iibcm5708 1000 base-T card. someone thought it might be “an over-enthusiatic 1Gb NIC”

(05 May ‘11, 09:29) gregg


3 Answers:

0

So, it looks like I found a fix: VMware_Networking_Speed_Issue

Cause: There is a bug related VMware network adapters or their drivers related to "segmentation offload" (TSO and GSO). I have heard one mention that the checksum feature may also have problems.

Solution: Turning off TSO/GSO on the guest OSes is the fool-proof solution. You could try installing a newer OS or try playing with the adaptor type (eg: E1000, VMxnet2) but that seems to work only some of the time.

Linux solution (worked for me)

apt-get install ethtool              # install ethtool if you don't have
ethtool -k eth0                      # view settings
ethtool -K eth0 tso off gso off      # disable tso and gso

NB: This linux fix does not persist through reboot. The article talks about how you might do that.

Windows solution (did not try, never had a problem with WinXP)

Network Connection icon -> Properties -> Configure -> Advanced Tab.

Look for "TsoEnable", "LargeSendOffload", "IPv4 TSO Offload" or otherwise and set it to 0 / Disabled.

answered 08 May ‘11, 09:08

gregg's gravatar image

gregg
1114
accept rate: 100%

edited 08 May ‘11, 09:11

1

When capturing on the same device sending packets, you will often see outbound packets greater than the MTU because the capturing software, tcpdump in this case, sees the packet before it's packetized on the wire. You may also encounter packets with less than the minimum number of bytes required because the padding hasn't been added yet.

As for the IDs, are you looking at the IP header's identifier field and trying to compare it with Wireshark's relative sequence numbers, perhaps? If you want to see the identifier, you can add a custom column to display it using ip.id as the display filter.

As for the fragmentation issue, what's the path MTU? Most likely it's lower than 1500 bytes, thus the ICMP destination unreachable message.

answered 05 May '11, 09:45

cmaynard's gravatar image

cmaynard ♦♦
9.4k1038142
accept rate: 20%

Yes, the MTU is 1500, which is why the ICMP does make sense.

The tcpdump ID I was going by: 10:52:04.077647 IP (tos 0x0, ttl 64, id >> 39774 << and in WireShark, Internet Protocol -> Identification. The two matched until I hit the 3 vs 1 packet. which after adding ip.id as you mentioned, showed what I was seeing in a more convenient place.

There's 3 virtual adapters across 3 virtual machines in play. - 1 on web server - 1 on wireshark - 1 on router

Wireshark records 3 packets, each 1500, coming out of the webserver, but the router only sees one big one coming in.

(05 May '11, 10:15) gregg

also, your comment about capturing on the same device as sending packets i think answers this:

[bad hdr length 16 - too short, < 20][|icmp]

(which i was originally ignoring since it didn't seem relevant to my real question) -- but it's good to know!

(05 May '11, 10:19) gregg

Where does the ICMP destination unreachable message originate from? Is it possible you are attempting to send a big 4420 byte packet, but due to PMTUD, as described in RFC1191, the VM responds with the ICMP destination unreachable message and the next hop MTU of 1500, then the fragmentation naturally occurs as a result, but the fragmentation is done at a lower layer so tcpdump never captures the fragments?

(05 May '11, 10:57) cmaynard ♦♦

The router is originating the ICMP message. Something is attempting to send a big packet. Specifically debian systems.

I started using netcat to just push data around for testing and if the debian systems are the source, then I get Fragmentation Needed ICMPs on nearly every packet.

This does not happen when the source is Windows (XP in this case)

If Debian is the source, the xfer takes a couple minutes.

If Windows is the source, the xfer takes 2-3 seconds.

I'm still working through this, but running out of ideas.

(07 May '11, 22:00) gregg

thanks for helping think about the problem!

(08 May '11, 09:20) gregg

1

Let me get this straight, you have a client (192.168.5.199) connected to the Smoothwall (gate) through a virtual switch vSwitch-X and the Smoothwall (gate) is connected with a second interface to the server (10.0.0.16) through vSwitch-Y.

When you try to open a webpage on the client, on the webserver you can see 3 outgoing packets in Wireshark, while tcpdump on the server interface of the Smoothwall shows you 1 incoming packet from the webserver. Or are you running tcpdump on the client facing interface?

The behavior is weird, but it looks like somehow one of the systems is trying to create jumbo frames out of the 3 separate packets. And smoothwall does not like it.

The "[bad hdr length 16 - too short, < 20]" part of the output is due to the fact that the ICMP message contains part of the original packet that caused the unreachable message. tcpdump is interpreting this payload and sees a packet that is cut in the middle of the tcp header (which should at least be 20 bytes long, but only 16 bytes are present).

answered 05 May '11, 16:17

SYN-bit's gravatar image

SYN-bit ♦♦
17.1k957245
accept rate: 20%

All of your assumptions are correct re:

.199 client -> vSwitch-X -> Smoothwall -> vSwitch-Y -> .16 server

WireShark is on a separate machine on vSwitch-Y

tcpdump is on the server interface of the smoothwall (vSwitch-Y facing)

In testing, I'm discovering that the direction does not matter. I also discovered that if the source is a debian system, the problems occur (ICMP on nearly every packet).

Windows systems as the source are very fast; no ICMPs

Destination machine does not matter debian->debian or debian->win (slow)

win->win or win->debian (fast!)

(07 May '11, 22:13) gregg

Also, if WireShark is on vSwitch-X: I see tons of

TCP Previous segment lost

TCP Dup ACK

TCP Out-Of-Order

(07 May '11, 22:17) gregg

thanks for helping think about the problem!

(08 May '11, 09:21) gregg