RST + ACK + CWR message, beginner needs help (FIX application)
Good day,
I am trying to debug problematic case with FIX message is being lost on regular basis. What I mean by this, is there are two applications (FIX client and FIX server). Those applications establish connection and start exchanging Hearbeat messages. When finally client application decides to send order message (this message is of longer size and can not fit in one packet, it takes 2-3 packets normally), connection drops with "An existing connection was forcibly closed by the remote host." message produced by both application's network api (Windows).
I have recoded pcap file
I wonder would could be possible reason, or if unclear, what should I review / check next.
Thank you for your help.
Update 1: Adding previous day pcap file as requested by Christian_R in comments.
Update 2: Adding both server pcap and client pcap files. It looks that checking client side pcap has something very interesting in it at around Frame #325. What does that "TCP Out-Of-Order" sent by client itself mean?
Interesting case. But can you trace a whole session: -Session setup - Heartbeat - start sending - Session Drop
I have some question marks in mind, especially the used IP Flags are not clear to me at the moment.
Unfortunately, I started recording this session too late and do not have all the messages since TCP connection is established. So even if I upload entire pcap file, all it would have is continuous set of Heartbeat messages coming to / from server / client. Do you think this will help?
Otherwise I can provide pcap file from previous day, where you can see the initial session setup, but not error. I must say that the error happens in most (but not all the days) we have some days (rarely though) when the issue does not happen at all. What is also worth mentioning is that if the issue happens, it tends to happen to the first order (longer) message of the day. All the rest order messages get delivered no problems after that (when the new connection and new session is setup). Let me know if the pcap file from day ...(more)
A better trace would help.
But in the meantime the session initiation of the old trace could help, too.
OK, I've updated the question with the link to the previous day pcap file. It has the messages since the connection was established until the first successfull order message comes.
"..When finally client application decides to send order message.." But in your first trace client didn't send anything, it's just getting RST out of nowhere.
Moreover this RST has jump in Seq. number, this is why the client issues Dup ACK. The second RST uses correct Seq.
What's your link/environment? Are there many middleboxes?
"....But in your first trace client didn't send anything, it's just getting RST out of nowhere.
Moreover this RST has jump in Seq. number, this is why the client issues Dup ACK. The second RST uses correct Seq...."
Yes, I agree with you. The seq number jumps and the root cause of that is what I am trying to understand.
It also seems that what I call client is what you think of as server (if I am not mistaken). To be precise 135.17 is what I call client, while 93.13 is what I call server (93.13 is listening on the port (5001), while 135.17 initiates connection)
The frame #6 is the frame in question. This is the frame I expect to be the one with the 'order message' in it. But it seems that the 'order message' frame(s) is(are) actually lost ...(more)
"....What's your link/environment? Are there many middleboxes?...."
The machines are physically located vary far away from each other. So I am not sure about 'middle boxes'. The machines communicate via ipsec site to site vpn.
"...A better trace would help...."
It always happens this way... The issue happens every day untill you start hunting for it. I did not happen today, so I will continue recording tomorrow. I will also try adding client side pcap file if it helps.
Ah, ok, I just assumed 93.13 to be a client because it has private range IP :) Wrong assumption. So this is server-side trace.
I guess jump in Seq. N happens because the packet from client gets lost in transit. Indeed client side trace would help very much 'cause we'll see whether this (these) missing request packet(s) existed on the network.
So probably the client sends "order" message (maybe even several times), gets no ACK and just resets the whole connection after timeout.
As for network itself ipsec always tells me "check for MSS/MTU values and ICMP blackholes on the path".
"RST+ACK+CWR" - actually I'd not check this on the first place, but after MTU behavior on client side. Probably ECN feature is ON on the client, therefore it tells about it this way. Try to disable ECN on the server, so it won't ...(more)
"...As for network itself ipsec always tells me "check for MSS/MTU values and ICMP blackholes on the path..." Thank you, we will try to look into this now.