Incomplete TCP Initial-Handshake

asked 2017-12-18 12:28:09 +0000

KwirkMiyagi
1 ●2 ●3 ●3

updated 2017-12-19 10:39:19 +0000

I am irritated about the following scenario:

I am running a MVC application on an IIS 10 Webserver. While calling the URI, it takes about 10 seconds until the application starts to get called (IDLE time). Having no real explanation for the occuring idle, I started to dig deeper using wireshark and stumble across the following phenomenon: The Initial TCP Handshake is incomplete (Missing ACK) 1. Client -> Server (SYN) 2. Server -> Client (SYN, ACK) 3. Client -> Server (ACK never reaches the server)

The topology is as described below.

Client -> Gateway -> Router -> Server1 -> Server2

Note: The Client is redirected to Server2 by Server1. Thus the page load time sums up to ~20seconds. The first 10s are reached by hitting Server1, and the next 10s by being redirected to Server2.

I used wireshark on client-side as well as on server side. The server resends the SYN,ACK tuple two times, after about 10s IDLE time, the connection is established nonetheless. Looking at the appropriate RFC, this behavior is normal (the ACK is sent by client indirectly while sending the data). The ACK never reaches the router, so where could it get possibly lost? And why does it get lost EACH time? Could it be some router (Mikrotik) setting, like No-ACK? Is the missing ACK the reason for the 10s IDLE delay?

Edit:

I updated the topology information above. I did anonymize the wireshark traces, they can be found at the following URIs:

Due to the anonymization process in TraceWrangler, the IP addresses vary from trace to trace in the following way:

Client IP: 192.168.248.249 <=> 172.23.147.181 <=> 192.168.201.209 <=> 10.206.108.221
Router IP: 10.194.30.227 <=> 172.17.84.111
Server1 IP: 172.31.124.208 <=> 10.100.24.4
Server2 IP: 172.20.78.56 <=> 192.168.204.149

You may use the following filters to get a clear glimpse on the initial handshake:

Filter Client: (((ip.dst ==192.168.248.249) && (ip.src ==10.194.30.227)) || ((ip.dst ==10.194.30.227) && (ip.src ==192.168.248.249))) && (tcp.flags.syn==1 ) || (tcp.flags == 0x0010 && tcp.seq==1 && tcp.ack==1)
Filter Router: (tcp.flags.syn==1 ) || (tcp.flags == 0x0010 && tcp.seq==1 && tcp.ack==1)
Filter Server1: (((ip.dst ==172.31.124.208) && (ip.src ==192.168.201.209)) || ((ip.dst ==192.168.201.209) && (ip.src ==172.31.124.208)) || ((ip.dst ==172.31.124.208) && (ip.src ==172.20.78.56)) || ((ip.dst ==172.20.78.56) && (ip.src ==172.31.124.208))) && (tcp.flags.syn==1 ) || (tcp.flags == 0x0010 && tcp.seq==1 && tcp.ack==1)
Filter Server2: (((ip.dst ==10.100.24.4) && (ip.src ==10.206.108.221)) || ((ip.dst ==10.206.108 ...

(more)

edit retag flag offensive close merge delete

Comments

Could it be some router (Mikrotik) setting, like No-ACK?

No idea, but if you have admin access to the Mikrotik, you can ask it to capture (sniff in their terminology) at both client-facing and server-facing interface simultaneously. The sniff files are standard pcap or pcapng files which you can then analyse using Wireshark.

This way you should see whether the client actually ever sends that standalone ACK to the wire/air and whether the Mikrotik forwards it.

sindy ( 2017-12-18 14:53:03 +0000 )edit

Can you share us the traces from client and server side? https://blog.packet-foo.com/2016/11/t...

Christian_R ( 2017-12-18 19:44:22 +0000 )edit

Thanks for your great suggestions, I followed the instructions and updated my post above with the appropriate traces.

KwirkMiyagi ( 2017-12-19 10:39:43 +0000 )edit

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2018-01-08 09:17:20 +0000

KwirkMiyagi
1 ●2 ●3 ●3

updated 2018-01-08 09:17:47 +0000

I have to apology for my late reply, but I wasn't in office the past weeks. I wish all of you a Happy New Year!

You won't believe it, but the problem got solved in the meantime, I don't really understand, why the following change in RouterOS ensures that the ACK is not lost anymore, and the application gets loaded in a hurry, but I am really relieved about it: Inside the RouterOS Route lists, the Gateway IP was entered as an IP address instead of an interface name / DNS name.

Do you have any explanation for this issue? Does the translation/lookup take so many seconds, and the ACK is ignored in the meantime? I always had a feeling, that it has to be an issue in the RouterOS, but I had no idea how to track it down. What a lucky coincidence that our admin was playing with the router tables, and asked me to check the loadup time again. Can this really be the only change? I was able to confirm in wireshark, that the ACK is not lost any longer.

edit flag offensive delete link

Comments

That could be. Maybe you have changed the interface settings or something. else. As I have told I had thought that there has been packet loss between the router and the gw.

Christian_R ( 2018-01-08 09:38:14 +0000 )edit

I'm a bit lost in your explanation what the actual change was. A gateway is normally an element in the machine's own subnet, so obtaining its IP address by means of DNS resolution doesn't seem like a useful idea as the DNS may not be accessible until the resolution is done. However, there are some resilience (backup) techniques which use chained routes which seem to be specific to Mikrotik, and there often a DNS server is used as a monitored address to verify connectivity via an interface. But in these arrangements it is pinged, not queried.

Now if an interface name, not a hostname, was set as route output, the situation is different - with interface name configured as the output of a route, the sending machine has to rely on icmp router discovery mechanism and icmp redirection when it needs to find an actual gateway element for ...(more)

sindy ( 2018-01-08 09:59:48 +0000 )edit

It's really that one configuration only, and it was probably due to the lookup process you have described ("icmp router discovery"). The gateway is not defined in the router's address list, instead the interface (of the appropriate subnet) was used as gateway, the routerOS allows to enter IP address, interface, etc., but recommends to enter an IP address explicitly. Could you explain, please, in which steps the discovery might happen and at which point the ACK gets lost? It's not quite clear to me, yet, still, I am really glad, that the huge lack is gone.

KwirkMiyagi ( 2018-01-08 11:46:24 +0000 )edit

As @sindy stated, this is not a Wireshark question anymore. Please contact the mikrotik support forum for RouterOS related questions.

Jaap ( 2018-01-08 12:10:22 +0000 )edit

Has already happened, Jaap, just thought to response to Sindy's comment.

KwirkMiyagi ( 2018-01-08 12:13:48 +0000 )edit

see more comments

answered 2017-12-19 12:44:02 +0000

Christian_R

2059 ●11 ●73 ●51 http://crnetpackets.com

updated 2017-12-19 12:47:05 +0000

Quite interesting Traces, I don´t know the infocrypt protocol here. But assuming the Gateway is also a infocrypt L7 Gateway, then it could be that the first two SYN ACK from the Server1 does not reach the GW. Can this be, true? I think the packet loss occurs between the router and the Gateway. Pöease check for Duplex Mismatches or error counters on the devices.

edit flag offensive delete link

Comments

Thanks for your analysis. You can ignore the infocrypt protocol, wireshark was just translating the client source port to infoprotocol, it's a random port number on client side. I only have access to the router, not to the gateway before the router (limited to our admin). I assume, too, that the packet is lost from gateway to router, but I have no explanation and I am not able to record this data loss. I tried to stream/sniff data from gateway, too, once (our admin activated streaming from gateway), but strangely, I was not able to see the initial handshake or any other communication with the client or router. Is that normal? (gateway-side) How am I able to make duplex mismatch or error counter checks? Is there a possibility on routerOS (Mikrotik Router)?

KwirkMiyagi ( 2017-12-19 13:00:40 +0000 )edit

Duplex mismatch can be spotted, if the switch or router counters are showing CRC errors.

Christian_R ( 2017-12-19 13:36:51 +0000 )edit

Not much a Wireshark question, but RouterOS CLI command is the following:

interface ethernet print stats

you would be interested in rx-fcs-error and all those tx...collision values.

However, should any issue on the wire be the cause, I would expect much more packets to be lost than just the ACK to SYN, ACK, and I would expect that one not to be lost systematically. Not that I would understand why two duplex-capable interfaces should ever lose packets - the crc (fcs) error normally occurs if two packets are transmitted simultaneously on shared media, which is not the case if duplex settings do not match between two duplex-capable ports.

sindy ( 2017-12-20 19:47:36 +0000 )edit

After answering the sub-question I've also had a look at your traces, so a few points:

if you trace an issue like this, it is important to take all traces simultaneously, because not only time relationship between the packets is an important hint for the analysis but also some information from the TCP header allows to find out whether a given transit element transparently forwards the packets or tampers with them; this is only true if the very same TCP session is present in all the captures
if anonymisation is necessary, I'd recommend to anonymize all the files together so that each unique address is translated to the same substitute in all of them, or use fixed mappings for important addresses
I've told you on purpose to sniff at all the Mikrotik interfaces simultaneously. The point is that if your capture comes from the server-facing interface, it ...

(more)

sindy ( 2017-12-20 20:23:22 +0000 )edit

Just followed my own advice and found out that it wasn't precise:

you don't need to sniff at all interfaces, you can list just those you really need
unfortunately, even if you sniff at more than one interface, the resulting capture file is a pcap one, not pcapng, so you cannot see at which interface the packet has been captured

sindy ( 2017-12-22 09:46:21 +0000 )edit

see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Incomplete TCP Initial-Handshake

Comments

2 Answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

Incomplete TCP Initial-Handshake edit

Comments

2 Answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

Incomplete TCP Initial-Handshake