The VXLAN encapsulation done by NSX-V adds an extra header to each packet, making full-size packets too large for ethernet networks with a normal MTU. Are you sure all interfaces that transport VXLAN encapsulated frames have their MTU size increased to (the advised) 1600?
Looking at the three sessions:
- Facebook: The MSS is 1392, which apparently is low enough to not create segments that are oversized
- Comvault: Used the default MSS of 1460, there is a 1460 byte segment that is not getting through, but in frame 48, the server tries a smaller (1024 byte) segment and that does get through. So the session is slow, but does work because of the server trying smaller segments.
- Google: Uses an MSS of 1412 and there are 2800 TCP bytes missing, this is exactly 2x (1412 - 12). The -12 becaus for the extra TS options in each packet. So it seems the 1400 byte TCP segment with all the encapsulation ends up too big for the network card in host B.
Next steps for me would be:
- Check the interface settings on the NICs of both HostA and HostB, pay attention to MTU and enabled offloading features and firmware/driver versions
- Make a packet capture of the VM when running on HostA, capture both in the VM and on the NIC facing the external network
- Do the seame for HostB and compare the 4 pcap files (and do please share again so we can have a look too :-))
