This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Checksum Offloading when can it be useful

0

Hi everyone,

I hope everyone is well. I am looking at a network issue at work and need some guidance. Wireshark is installed on the target client machine and doing a number of tests from ping external sites, browsing the web and downloading files and accessing internal files servers and copying files .

My capture has come back with a allot of issues in the experts panel from Malformed Packet DCERPC which i presume isn't that bad as it only happened on in the siz minute capture, 13 Ack segments that wasn't captured all through the 6 minute capture. To 62 retransmissions. and duplicate acks again throughout the whole 6 minutes.

I am seeing allot of retransmissions, from what little i have learned these past 2 weeks .... with these ACKED unseen segments could allot of these packets be malformed and by me disabling the checksum option in wireshark it wont pick out the same errors either?

I am following Lauras checklist . Installed wireshark on the problem device and it seems like it is a good capture i.e no packets dropped. Syn and Syn-Ack times are positive fast (0.006 ms from syn from client. 0.01 -SYN-Ack) I ma now going through stats to find hte biggest conversations and then use my golden graph rule etc.

I really like taking the knowledge I am getting from books I have read so far and trying to apply to a real network:)

Thanks again as always for your time and help.

cheers

Jazz

The current arguments between the factions is that data transfer between 2 hosts on the same subnet is great but as soon as the traffic exits to go to the data centre via the MPLS and then further routed out via the internet breakout in a different data centre the additional overhead on these packets is causing issues by exceed any MTUs.

I for one am just keeping out until i find something concrete in the logs :)

When is the right occasion to reenable the checksum feature?

Cheers

Jazz

asked 24 Apr '14, 07:25

yoyomonkey's gravatar image

yoyomonkey
0226
accept rate: 0%


One Answer:

2

I would not worry about the DCE/RPC Malformed Packet messages. DCE/RPC is not always well documented and there are many many flavors of it, so the Wireshark dissector sometimes gets a little lost.

"Segment not captured" means that you may have had packet loss, or it may be dropped packets. Have you check the drop counter to see if it is greater than zero? "Acked unseen segment" is a clear indicator for dropped packets, which means your capture is too slow to record all incoming packets. 62 retransmissions in 6 minutes is not a lot, and the golden rule of evaluating retransmission impact applies, which is "how much time did it cost to recover from packet loss? And is that delay a problem?"

I also usually advise against installing Wireshark on the problem machine, because the captures are not as exact as they could be. It is much better to capture very close to the problem machine, but with a dedicated system. As soon as you do that you can also reenable the checksum feature.

Edit: By the way, what "checksum feature" do you mean exactly? In Wireshark, or in the OS? Please do not tell me you disabled Checksum Offloading in the OS... ;-)

answered 24 Apr '14, 07:53

Jasper's gravatar image

Jasper ♦♦
23.8k551284
accept rate: 18%

edited 24 Apr '14, 07:57

Hi Jasper,

Thank you so much for the helpful reply.

You mentioned about doing a capture as close to the problem device as possible. The issue I am looking at is everyone on the work subnet is getting slow downloads approximately 700kb/s going via a scansafe proxy which is situated in our seperate datacentre. If i RDP on to a machine in the datacentre i get 6MB downloads going through the same proxy machine. the datacentre and office networks are seperated by an MPLS. When i use a machine in the office subnet and go via a proxy it is 700kbs but if i go straight out avoiding the proxy I get 6MB download. If i download a file from the data centre i get 8mbs.

Allot of fingers have been pointed by different people so I am just learning wireshark and using this as experience.

As I believe it is not the machine that is the problem. Task manager on the device shows no bottleneck. Would you recommend I still use a seperate dedicate device to capture network traffic.

I have done all captures using tshark instead of wireshark?

If I should use another device can anyone recommend any hardware Taps that would be good enough to do the job but not cost the earth?

Thanks again

Jazz

(24 Apr '14, 13:38) yoyomonkey

You don't need to use Taps for this, a SPAN port is good enough in most cases. Just use a SPAN with another PC that is only recording what the other devices do and you're doing a better capture than directly on the problematic system.

Capturing using tshark or Wireshark doesn't matter, both are running a dumpcap process to do the actual capture.

What you need to do is to get a capture of the slow transfer and determine what is slowing it down. For that you need to understand how TCP works, especially the TCP window, and you need to determine if packet loss is a problem. Then compare what you found with a fast transfer (without the proxy) to see if it gets better.

Ususally, if speed is much less than expected there is problem with packet loss, TCP windo size, or buffer overruns.

(24 Apr '14, 13:46) Jasper ♦♦

Briliant thank you so much Jasper for clearing that up I thought as it is every machine in the network experiencing the slow internet download speeds via connecting to a proxy in the MPLS and when it goes direct it is fast by capturing on the device both scanerios I could work it out but i see your point and by intercepting the traffic instead of possibly attributing complication by running it on the same device.

I got the capture and from reading the 101 and just finishing the troubleshooting books ( Now on chapter 2 on the Network Analysis Manual :))

I was looking at the syn and Syn-ack times to get round trip time from the client. I disabled the TCP checksums on wireshark as well as look at the TCP Delta times between packets of hte same conversation. I am finding fast tcp handshakes but i also find the acks unseen segment.

I will get a spanning capture done tommorow and go back to Laura Chappells checklist and see if i can find everything:)

cheers again Jasper

Jazz

(24 Apr '14, 14:53) yoyomonkey

Pro Tip: to get RTT you should look at the full handshake time, from SYN to ACK, not just the SYN/ACK. That way it doesn't matter where the capture device was recording the traffic. SYN to SYN/ACK requires you to capture very close/on the client.

If you see ACKs of unseen segments your capture device is too slow. You need to fix that first, or you'll keep wondering if packets were there or not at all. With experience it is quite easy to argue what the truth was, but for beginners it is better to make sure the capture has no dropped frames.

(24 Apr '14, 15:00) Jasper ♦♦

thanks for the tip Jasper and your time. I have learned so much these past 2 weeks and your help and time is greatly appreciated :)

So that i have got this right in my head.

If i install wireshark on the same device that is experiencing the slow down i would not be able to to disprove that wireshark itself was the cause of the issue of unseen segments however if i span a port with a dedicated laptop with wireshark installed and i still see unseen segments I would be a better position to say that they could be a faulty switch dropping segments etc? As my recording device is totally seperate?

I guess what i am trying to understand is there any other reason for Unseen segments (in the middle of a capture) apart from a slow capture device? or is it ONLY a slow capture device.

I see many duplicate acks and retransmissions zero windows etc and i guess it will only be from experience and going through more captures until i gauge a genuine network error or an error which i have introduced from a bad capture:)

Thanks again

Jazz

(24 Apr '14, 15:35) yoyomonkey

Wireshark is not the problem if it runs on a system that is part of a problem, it is just that the measurements get "distorted" and may become misleading, especially for beginners. Also, other software like local firewalls, VPNs and tunneling devices may lead to incorrect results, too. If you run Wireshark on a dedicated laptop it is just recording what really happens "on the wire" (almost) without any bias.

Don't bet on faulty switches, they are pretty rare as a problem source by now and it is usually not necessary to prove problems like this with Wireshark. Instead let the switch admin take a look at the port error counters, which is much easier.

Yes, there may be other reasons for ACKs to unseen segments other than a slow capture device. It can happen when you capture on a link that is part of an asynchronous route, which means you see only packets in one direction - but that is pretty obvious because the other direction is completely absent. Normally, "ACK for unseen segment" is caused by a slow capture device. The reason can be a slow HDD, or a network card that can't cope with the speed of the packets coming in.

Zero Window is a warning sign, because it means that the device signaling it is experiencing problems with the load of incoming packets. You need to check how fast the device recovers from that situation (e.g. if it takes just a few milliseconds to recover it may be not that much of a problem), because a long delay until a positive window size is advertised can lead to a big drop in throughput.

Word of advice: don't wait for Wireshark to tell you "this here is the exact problem!", because it won't. In problem situations like yours you have to isolate the TCP session and find the places where it gets into trouble (e.g. filtering on "tcp.analysis.flags"). Then determine if the problems are critical or not. Best way to do this is to determine how much delay each problem caused. Microseconds or a couple of milliseconds may not be a problem unless showing up very often (and adding up), seconds are.

(24 Apr '14, 16:12) Jasper ♦♦

That is amazing. Your help and answers have been really eye opening for me Jasper. I guess my biggest mistake being a newbie is expecting Wireshark to tell me what the exact problem is and I am blindly accepting it where I should be spending a bit more time understanding what wireshark is seeing and drawing my own conclusions as to whether it is of impact to the user. I have started to look at logs and use a tcp delta columns, the golden graph and arrange by highest delay. I then ommit any RST ACK and FIN,, keep alive packets and also ones with user interaction involved etc. I hope that as time goes by i start to become more intune and see quirks allot easier one thing is for sure I am really enjoying the whole learning process:)

Thanks again Jasper

Jazz

(25 Apr '14, 00:29) yoyomonkey

No analyzer I ever saw can tell you exactly what the problem is, because there are many symptoms that have other explanations which are not a problem at all. It's all in the context.

I have to admit that network analysis often requires a lot of knowledge about protocols as well as a lot of experience - which is kinda hard to get because you can't "learn" experience from books - you need to get your hands dirty, and then try and fail and try again. All of us did it that way I guess :-)

Oh, and you could join us at the Wireshark conference in June. Tons of experienced presenters and attendants to talk to and learn stuff from ;-)

(25 Apr '14, 07:37) Jasper ♦♦

Thanks for the advice and the positive encouragement ....I forward to making more mistakes and getting my hands dirty.

I have been working on the issue and it seems now we may have a fix with the proxy in the datacentre that was giving our end users a slow experience in our office.

We built another proxy with the same software but installed on windows 2012 not 2008. Everything was kept the same even the proxy port etc and now connecting from the office our speeds are allot faster.

I am getting captures from the two machines and comparing the 2 to see if anything sticks out . Is it how some weird caveat with scansafe on a 2008 R2 VM ... is it the 2008 r2 with VMTools and a specific configuration or something else but as everyone else is celebrating I am going to continue until i find something conclusive and try and find out why :)

I would love to come to wireshark 2014 this year but with my job still running till august and money being tight I won't be able make it this year :( but these last 2 weeks of reading and working through the 101 book and troubleshooting and now on the analysis and the help i have recieved here from yourself and others in the community I hope to go to the next one and if there is a UK conference or meetups or other UK Wiresharkers swimming about please do contact me:)

I apart from this forum is there any other recommendations ...

I wont be able to afford to go out to wireshark this year but would be able to pay for the access pass. Can anyone recommend it and how valuable is it in my learning and preparing for the wcna in the future?

Any other resources? some people mentioned the tcp book and nmap etc.

I am doing small 10-20 minute captures on my home network ...laptop wired and wireless i am also loading up lauras logs and looking at syn syn-ack and ack tcp deltas to look for lags etc. tcp analysis flags etc basically usingg laura checklist as a starter :)...all rather complicated but very interesting :)

will the all access pass have videos from wireshark conference this year on it .

Anyway cheers again for aall your help.

Cheers

Jazz

(25 Apr '14, 13:14) yoyomonkey

It is quite possible that an older VM (I guess it's a VMware vSphere cluster) is not as performant as a newer VM. When describing problems it helps to mention that the system in question is a VM, because it adds a couple of factors to the situation that you don't have with physical setups. E.g. CPU/RAM/disk/network is shared between multiple VMs which can lead to bottlenecks, with disk I/O being quite relevant for proxies in my experience. Also, check if the new VM has "better" network adapters, e.g. VMXNET3 instead of Intel e1000, which can make a huge difference.

No worries about Sharkfest 2014, it was just an idea and I think you would like it. But money is an important factor, of course. I don't know about UK meetings, but here in Germany there is no such thing I'm aware of since it is a rare topic and it seems not many people really do network analysis. Maybe there will be a European Sharkfest at some point though. Riverbed does Wireshark Roadshows every once in a while, and I guess they also run them in the UK, so keep an eye out for those.

The TCP book (I guess they mean "TCP/IP Illustrated Volume 1" by Stevens) is always good to have. I wouldn't bother about nmap too much at this point since its focus is more in the security area and it is not that useful in troubleshooting systems.

Other ressources worth mentioning are LoveMyTool (http://www.lovemytool.com), Hansangs videos at http://blog.wireshark.org, and Tony Fortunatos Wireshark tips (usually seen on LoveMyTool/Youtube). You could also check out my blog at http://blog.packet-foo.com, even though I have to admit I'm too busy at the moment to write new posts :-)

Oh, and you don't need the all access pass to get the videos from Sharkfest; they are posted for free on Youtube and available via LoveMyTool and http://sharkfest.wireshark.org/retrospective.html

(25 Apr '14, 14:00) Jasper ♦♦
showing 5 of 10 show 5 more comments