I'm trying to dig into an issue I'm seeing on a number of systems in my environment. It's repeatable in a number of different areas, which makes me think there is something going on in the network equipment or NICs, but I'm trying to build up some evidence to help narrow the search. I get expected throughput when running throughput tests with netperf or iperf when running from a 1 GbE or 10 GbE connected client system to a test host that is connected via 10 GbE. If I choose a host that is connected to 1 GbE or switch to a NIC that is 1 GbE on the same host, I get terrible performance. i.e.. 900 ish Mb/s on the good test..... down to 10-50 Mb/s on the bad. I used tcpdump to capture traffic data and used Wireshark to examine. I'm a newbie with Wireshark and a relative newbie to network traffic analysis. I've attached an image of the wireshark 'expert info' summary from the poor performing test case (10 second test). Compared to the good performing 10 second test case, this trace has what appears to be alot of duplicate acks and retransmissions. In the good test case, I think I see perhaps at most one dup ack for packets transmitted. I'm asking here for confirmation of that assumption. And is there anything I can pull out of this trace to find a clue to what might be causing the problem? I can make this happen on several different systems, so it seems unlikely to be any one port/sfp/nic issue... but perhaps a bank of ports, or some other configuration issue could be causing it. In an effort to learn more instead of nag my Network peers, I thought I'd ask here first. Any advice for this Newb is much appreciated ;-) -lp
asked 24 Sep '14, 07:01  luverofpeanuts edited 25 Sep '14, 06:03 |
One Answer:
Ok, well then: It's very difficult to troubleshoot from screen shots, so we would be better able to help you if you uploaded an actual trace file somewhere and then provided a link so we could look at the real packets. Cloudshark is a good choice, but Dropbox or Google Drive would do as well. Having said that, duplicate ACKs and retransmissions are a sign of packet loss, so some device along the path is dropping packets. You'll need to move Wireshark along the path and capture in different places to see what device is dropping packets. answered 24 Sep '14, 09:09  Jim Aragon Thanks for the reply. When I realized I couldn't attach a file, I decided to post the screen shot first. I did just update the post with a link to my raw tcpdump file. ;-) (24 Sep '14, 09:52) luverofpeanuts |
Not sure if you're aware of this or not, but I can't download your trace because it's been downloaded too many times which has reached a bandwidth limit threshold. Perhaps you want to upload it to cloudshark for others to see. If you're concerned about anonymity, I just stumbled across a www.tracewrangler.com mention in another thread.
That's strange. I'll delete from google drive and put on cloud shark. Thanks for pointing that out.