SSH Connection randomly drops (Palo Alto FW in between)
An SSH connection to a particular server drops randomly (usually 20-60 seconds after login). Between the client and the server is a Palo Alto firewall with SSH decryption disabled.
What I tried so far
- regenerated ssh keys on the server
- added to server config: ClientAliveInterval 30 ClientAliveCountMax 5
- added
ServerAliveInterval=10to ssh command - added
ServerKeepAlive=trueto ssh command - tried various ssh clients
Nothing worked so far. Notice the debug3: send packet: type 80 and debug3: send packet: type 1 messages just at the moment before/ after the connection is dropped. The firewall logs the SSH session and the termination reason is "tcp-rst-from-client".
I did a packet capture from within the firewall. Palo Alto allows to capture four different flows:
- drop —When packet processing encounters an error and the packet is dropped.
- firewall —When the packet has a session match or a first packet with a session is successfully created.
- receive —When the packet is received on the dataplane processor.
- transmit —When the packet is transmitted on the dataplane processor (from here)
It seems like the client sends a TCP RST message to the server. I am not an expert on analyzing such traces and hence would appreciate any support from you experts. I would like to append the capture to this thread, however it seems like my karma is pretty bad ;)
Thanks in advance.
Post your capture on a public file share, e.g. Google Drive, DropBox etc. and post a link to it back here as a comment.
Hi Graham, thanks for advice. Here you go: https://we.tl/t-zNf4VJ45YU
A network diagram would be of a great help here because there is an asymmetric path involved together with FHRP protocol (check MAC addresses).
Client resets a connection probably because of timeout. There is a gap in SEQ# between last client's ACK and RST which means some client's packets were lost before capture point.