Ask Your Question
1

fortigate ip connection

asked 2023-02-23 05:56:22 +0000

quest4answer gravatar image

updated 2023-02-23 06:17:55 +0000

Hello few of our application which are hosted on our main DC are abruptly terminating their session. there is a FW in front of those application. branch and DC are connected via SDWAN. FW is fortigate and throwing "IP Connection error" for each abrupt disconnect of those application https://community.fortinet.com/t5/For... it seems like reply packet is missing so FW is disconnecting the session with connection error

i put sniffer infront of one of the application server and saw some issues starting from packet 29 please see a link below for capture. https://drive.google.com/file/d/1qxJh... around 15:24.8 i see ip connection error on firewall logs which sort of reflects the problem starting from packet 29. the interesting thing is ,, this is clear server-rst from packet capture standpoint but fw reports as ip connection error..not server -rst

packet 26 is graceful fin/ack...initially i thought that packet 27 and 28 never makes it to server and thats why packet 29 is the reset but when i check the ack on packet 29 , i noticed that it is acking packet 28 with seq 2516 .. so i am really confused..And after packet 29 whole capture is problmatic. ...also noticed that 119 seconds its commonality for some sort of timer..i checked few of them and they all have this 119 seconds timer and then have same behavior .. one strange thing is packet 31 has strange ack no.. not sure how it shows up..because couldn't relate to other side.. and then its keep doing it.. one thing for sure clients fin-ack do not get ack from server .. but instead we see weird sequence no. packet going out. may be that is reason FW is erroring it.

i have this behavior whenever there is a ip connection error on FW. I am trying to find out what is the root cause. but from this capture it seems like we are loosing some packet but couldn't pin point exactly. does anyone has any thoughts. thanks

edit retag flag offensive close merge delete

Comments

Looks like a "half-closed" connection - TCP client responses with no FIN

It would help to have a capture on both ends to see if the client is sending the FIN and if so, is the firewall holding the connection half open.

Chuckc gravatar imageChuckc ( 2023-02-23 15:21:34 +0000 )edit

yeah agree .. make sense to have capture both sides. but its not possible. but if you look at the sequence no. based on that it seems like fin-ack from client is not getting responded from server. also server is responding with some strnage ack. what about that.. any thoughts on that

quest4answer gravatar imagequest4answer ( 2023-02-23 16:00:32 +0000 )edit

Normal client/server stuff here.

No. Time    Source  Destination Protocol    Length  Info

22  1.355066    10.10.10.1  192.168.0.1 TCP 943 443 → 39078 [PSH, ACK] Seq=7735 Ack=2516 Win=261632 Len=889
23  1.520244    192.168.0.1 10.10.10.1  TCP 60  39078 → 443 [ACK] Seq=2516 Ack=7735 Win=46592 Len=0
24  1.559343    192.168.0.1 10.10.10.1  TCP 60  39078 → 443 [ACK] Seq=2516 Ack=8624 Win=49152 Len=0

8 seconds of inactivity - server want to close the connection.
There is no client FIN in frames 25-28.

25  9.366601    10.10.10.1  192.168.0.1 TCP 85  443 → 39078 [PSH, ACK] Seq=8624 Ack=2516 Win=261632 Len=31
26  9.366659    10.10.10.1  192.168.0.1 TCP 60  443 → 39078 [FIN, ACK] Seq=8655 Ack=2516 Win ...
(more)
Chuckc gravatar imageChuckc ( 2023-02-23 16:14:45 +0000 )edit

yeah it seems like packet 29 is rst-flag from sever due to client not responding from its side with fin-ack.. in that case i would see server-rst on FW but i dont.. which is strange but also what about those weird ack from server

quest4answer gravatar imagequest4answer ( 2023-02-23 17:01:12 +0000 )edit

Which frame(s) are "weird ack from server"?

Chuckc gravatar imageChuckc ( 2023-02-23 18:22:37 +0000 )edit

2 Answers

Sort by » oldest newest most voted
1

answered 2023-04-14 11:05:51 +0000

BobNetD gravatar image

It appears that the server has an idle timeout of 8 seconds and closes the connection graciously by sending close-notify Alert and Fin packets. The client was probably using a browser and most, if not all, browsers maintain a pool of connections. This connection had been returned to the pool in which case there was no application thread to handle the half-closure and direct TCP to complete the closure. It further appears that the browser’s pool manager has an idle timeout of 120 seconds, and the server has a Fin-Wait timeout of 120 seconds which produces the coincidence of a Reset from the server and a Fin flag from the client.

It also appears that the Resets from the server have not been relayed to the client. The client repeats its Fin flags, seeking an ack, but the server always responds with a Reset which is not relayed to the client. The subsequent Resets are constructed with different sequence numbers because the server is now in a different state and their purpose is not to close the connection but to indicate that the connection no longer exists. It is strange that the firewall will relay client Fin packets but not server Reset packets.

Problems like this are very common yet often produce highly varied and bewildering results. The cardinal rule for connection pools is that they must have an idle timeout that is shorter than the timeouts of all the relevant servers. This ensures that connections are never left in a half-closed state. Weird problems arise when firewalls and other devices along the path have shorter timeouts and terminate half-closed connections in their different ways, usually without signalling the termination by sending a packet to either the client or the server.

One solution in this case is to increase the idle timeout of both the server and the firewall to more than 120 seconds. A better solution might be to reduce the browser’s idle timeout to less than 8 seconds, assuming that the firewall’s timeout is greater. It’s a salutary lesson for all network managers to ensure that the timeouts in all devices are set appropriately and not left with their default values.

edit flag offensive delete link more

Comments

A clue that the client isn’t receiving the server’s Resets is the doubling of the time between successive client FIN packets. The timing has the flavour of retransmission timeouts, indicating that the client never saw any response to each FIN.

Philst gravatar imagePhilst ( 2023-04-14 19:18:11 +0000 )edit
0

answered 2023-02-24 10:33:13 +0000

SYN-bit gravatar image

yeah it seems like packet 29 is rst-flag from sever due to client not responding from its side with fin-ack.. in that case i would see server-rst on FW but i dont..

That would be the case if the server sends a RST without first sending the FIN, but since it sends a FIN first, (that is not answered by a FIN from the client) it could possibly log this differently. An "IP connection error" might be the way it is reporting FINs that do not get a FIN back. You would have to check with Fortinet Support.

In any case, for an HTTPS connection, it is really weird for a client to keep the connection open after the server has sent a FIN (indicating it will not send any more data after that point). Do you know what kind of client is doing this? It seems the client waits 120 seconds to send the FIN after receiving the FIN (it is not a response to the servers RST, as that would mean you would have seen ~165 ms delay between the two packets), that is also weird behavior. So I would mark this as a problem on the client side (or in any infrastructure between the client and the capture point).|

The weird ACKs seem cosmetic to me, somehow the ACK of the packet resulting in the RST is kept, but since the ACK flag is not set, it should not treated as "insignificant". Some security products might trip on this as it is not common, but AFAIK the RFCs don't explicitly say the ack field should be zero when the ACK flag is not set.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2023-02-23 05:56:22 +0000

Seen: 3,565 times

Last updated: Apr 14 '23