 | 1 | initial version |
Again, thank you very much for your response. It did point me in the right direction. I believe I have solved the issue. For sake of documentation I will write about my solution.
During the IP handshake both endpoints send each other their MSS, which is the lesser of the receive buffer or the result of MTU - 40. Both endpoints will send packets to each other at size of the smaller MSS. This calculation is based on NIC's MTU. So, if the NIC's MTU is 1500 the resulting MSS is 1460.
In my set up, for some reason PMTUD is reporting the WAN MTU (MTU of next hop) for the router as 54 bytes less than the configured MTU. So, if the WAN MTU is 1500 than the MTU of the next hop is reported as 1446. When the server sent a packet of 1460 it became 1500 after adding the header and footer to the packet. This new size is greater than the MTU of the next hop and required fragmentation.
The problem I was experiencing was that the server would not fragment when informed of the need to fragment from the ICMP message until it tried sending the packet 3 times. The fourth time it would sometimes fragment to 576. When it did not fragment the connection would be reset by the server because the client would never receive the expected packet, causing the computer to not complete the authentication conversation.
Changing the MTU of the NIC required the following two commands at the CMD prompt:
This is a very crude explanation due to the fact that I'm running out of available characters. But the take away is that the NIC's MTU has to be less than the Router's MTU in order to allow for VPN headers and avoid fragmentation.
