Win7 workstation -> LAN -> ASA -> Cisco ASR -> DMVPN -> ASR -> Palo Alto -> Nexus -> NetApp

We are experiencing the symptoms described in the title. This is not new, it predates me, and it happens at multiple spoke sites in our DMVPN. Each vendor just seems to point the finger at the other with no real data reinforcing their point. Cisco has cleared any real issues at the hardware level.

alt text

alt text

alt text

asked 28 Sep '16, 11:57

wdurand's gravatar image

wdurand
50117
accept rate: 0%

Hm hard to tell, with the info you gave. Could share us a tracefile at public accessible palce. You could use https://www.tracewrangler.com/ to do some anonymisation.

(28 Sep '16, 12:16) Christian_R

Let me know if this works, if not I'll try something else. thanks.

https://www.dropbox.com/sh/o7a5bigcg9aixmm/AADxJRAUGxlwARnG1m0LJVKGa?dl=0

(29 Sep '16, 04:31) wdurand
2

While writing retransmissions can be seen. While reading there are less Bytes in flight. I think it could be a window size problem, but to be more reliable we must see the handshake which is not included in the trace. So we need a new one which has the handshake included.


Update: So I have deleted my first answer, because I spotted some thing that I neeed to investigate. The SMB response goes up in worst case up to 2 sec. And I will connvert this answer into a comment, because at the moment it is no answer for me.

(29 Sep '16, 10:13) Christian_R

I uploaded a regular Windows file copy of a 1G file to the netapp, C to H, then I did a copy back to the workstation, H to C.

(29 Sep '16, 10:48) wdurand

Well as I told without the three way handshake. The trace is not much worth. So maybe you need to disconnect and reconnect the network drive while capturing. Or you could try to reproduce the problem with FTP or SCP???

(29 Sep '16, 11:41) Christian_R

Okay, so I tried the copy another way. I closed all sessions on the netapp that were associated with my userid.
cifs session close -node SCORSAN01 -windows-user I then started a pcap session, mapped the H drive and transferred a 200MB file up. I repeated the process after disconnecting again and reading the file back to the workstation. Please see captures.

(30 Sep '16, 07:48) wdurand

Yes the tracefiles are working. So one question is why 198.207.240.111 only a MSS of 1360 advertises?

(30 Sep '16, 11:56) Christian_R

I have updated my answer.

(30 Sep '16, 15:19) Christian_R

That is set on the tunnel interface for DMVPN.

interface Tunnel0 ip mtu 1400 ip tcp adjust-mss 1360 load-interval 30

(03 Oct '16, 06:25) wdurand

Ok then it is like I have thought the mss is correct.

(03 Oct '16, 06:56) Christian_R

Well you try to tune some parameters at both sides, as I think the limitaion is either the receive or the congestion window. But you really should do this very carefully.

Some tuning hints could be found here: http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574

and here: http://www.speedguide.net/articles/windows-8-10-2012-server-tcpip-tweaks-5077

and here: http://smallvoid.com/article/winnt-tcp-slow-start.html

(03 Oct '16, 12:28) Christian_R

An excellent question. We will discuss this at Sharkfest Europe!

(19 Oct '16, 06:30) packethunter

Which Palo Alto Networks Firewall is used? Is there a Security Profile in place for the Firewall Rule which matches the session?

(21 Oct '16, 06:02) Markus_W

What Bandwidth does the WAN have?

(21 Oct '16, 07:01) Christian_R

The client does increase the amount concurrent read request over time in a linear way. In my test scenario I haven't seen this behavior. Maybe it is due to your iRTT of 23ms? But I don't know for sure as I am not a SMB expert. Maybe You can try to tune the next parameter, but you should do it in very carefully way. Because their might be a very good reason why the default is 0.

Client

DisableBandwidthThrottling

HKLM\system\CurrentControlSet\Services\lanmanworkstation\parameters\(REG_DWORD)

The default is 0. This setting is available starting with Windows Server 2008 SP2. By default, the SMB redirector throttles throughput across high-latency network connections in some cases to avoid network-related timeouts. Setting this registry value to 1 disables this throttling, enabling higher file transfer throughput over high-latency network connections.
(30 Oct '16, 01:42) Christian_R
showing 5 of 15 show 10 more comments

At the end of my talk about SMB2 during Sharkfest Europe I have invited the attendees to look at this post to get a feeling for the protocol. I guess, people are interested in SMB2 since the view counters went up over the last week.

My analysis can be found on Jasper's blog as it became too big for this website.

Any comments, add-ons etc are very welcome.

link

answered 28 Oct '16, 12:33

packethunter's gravatar image

packethunter
1.8k71344
accept rate: 5%

My 2 cents:

C_to_H_200MB: Trace taken on the sender side. Throughput is linearly increasing till 240Mbit/s at the end, like the receiver window size, growing all along. Windows size is not filled from sender point of view.

H_to_C_200MB: Trace taken on the receiver side. Throughput grows linearly (max of 180Mb/s), but is stopped each time there's a retransmission, sender seems to be very sensitive to that.

I'd make the test with a Linux & macOS client.

If changes, I'd check and disable some "Slow start" / Heuristic features of Windows:

http://betanews.com/2011/01/20/use-hidden-windows-tweaks-to-speed-up-your-internet-and-network-connections/ https://www.duckware.com/blog/how-windows-is-killing-internet-download-speeds/index.html http://www.speedguide.net/faq/windows-scaling-heuristics-changes-my-tcp-autotuning-339

Else, throughput is high, and RTT of 30ms, I'd try to open more both RX windows size and TX buffer at the beginning of the TCP connection to see if transfer throughput is no longer linearly growing.

link

answered 27 Oct '16, 04:58

TomLaBaude's gravatar image

TomLaBaude
66131523
accept rate: 66%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×50
×14
×8
×2

Asked: 28 Sep '16, 11:57

Seen: 912 times

Last updated: 30 Oct '16, 01:45

p​o​w​e​r​e​d by O​S​Q​A