TCP RST & RST, ACK question

asked 2019-05-17 20:56:11 +0000

cpocket
1 ●2 ●5 ●6

Troubleshooting why IP trunks between two pbx's randomly drop. They talk on port 1067. There are no Firewalls between the sites its a private L2 circuit between them. I captured packets at 4 points right at the PBX and WAN rtr and compared all 4 captures. I want to know if I'm understanding the packet with RST, ACK in it.

PBX A 1.1.1.1 shows a log at 17:58:53 that IP trunk link failed between A and PBX B (2.2.2.2) WS logs captured on PBX A at 17:58:54 there's a RST, ACK packet sourced from PBX B destination PBX A. This packet suggests to me that PBX B 2.2.2.2 is acknowledging a TCP RST send from PBX A correct? However, There is no RST packet sent from A prior to this 17:58:54 time stamp.Furthermore the TTL of this packet is 254 which tells me it truly didn't come from PBX B. Then two packets later 17:58:55 there is an actual TCP RST generated from PBX A heading to PBX B. I can follow this packet all the way to PBX B. Then when I look at my PBX B logs It shows IP Trunk Failure at 17:58:54.

So how does this, what looks like to me a fake RST, ACK packet sourced from B to A get generated? What other clues could help my pin point if this is truly a network issue or PBX application issue?

Thanks

This packet has a TTL of 254 so I know it didn't truly come from PBX A and there's no TCP RST packet sent from A in my other captures for B to

edit retag flag offensive close merge delete

Comments

There is no ACK to a RST, only to SYN and FIN. Why is TTL 254 so telling for you? Is that different from TTL received on other packets? Does the TCP connection have keeps-live messages, preventing the intermediate boxes to timeout the connection?

Jaap ( 2019-05-18 04:26:48 +0000 )edit

Hi Jaap,

I believe the TTL of 254 is important because PBX B is 4 hops away. Assuming the RST packet came from a device 1 hop away this would be coming from a Cisco 3850. PBX A is directly connected to that same 3850. The TCP connection does have keep-alive messages. It must be a timer around 10 seconds as I see this pattern a lot within my captures. Traffic between the two devices is going smoothly I will see TLSv1.2 application data back and forth and an ACK packet typically from PBX A then 9.63 whole seconds PBX B will send a Keep Alive to A and A will respond in 0.000185000 then after A sends the Keep Alive ACK PBX A doesn't generate another packet for 6 whole seconds. This keeps happening then the last series of packets goes like this.

PBX A ...

(more)

cpocket ( 2019-05-20 13:08:30 +0000 )edit

add a comment

answered 2019-05-19 13:07:43 +0000

SYN-bit

18528 ●9 ●343 ●255 https://SYN-b.it

From your description one can deduct that there is an intermediate device generating the RST impersonating PBX B. This can be concluded by the fact that you don't see this packet in the trace at PBX B and also because the IP TTL is different. You can look at the ip.id of packets from B and if they slowly increase and the ip.id of the RST is way off, that's another indicator for it being sourced by a different system.

As for the ACK bit, this bit is set when the system sending the packet took care of generating a proper ACK value. The ACK bit set means "ACK field is relevant".

Was there an idle time in the connection before the RST from B to A was generated? In that case it might be a session based device that hit an idle timer. If so, you can either find that device and change the idle timer for the protols that are used between PBX A and PBX B. Or you can use a Keep-Alive mechanism, either at the TCP level or at the Application level.

If there was no idle periode before the RST, was there a packet being sent by PBX A just before receiving the RST? If so, it might be an IDS or IPS that has a (false) positive on it's detection rules and it was configured to actively close the connection on this detection. In that case, find the IDS/IPS and check/update the detection policy.

In either case, it would be interesting to know which system was responsible for sending the RST and it should not be difficult to find as the TTL is 254 which means it can only be one hop away (unless you have some MPLS or other tunneling in place, then of course it could be on the other side of the link).

edit flag offensive delete link

Comments

Hi Syn-bit,

Thanks for your comments. I believed I answered some of your questions when I responded to Jaap. To confirm what he said then I should not see a RST packet with RST, ACK in the info field? There is no IDS/IPS between these two devices. You are correct about the ip.id field on PBX A capture most the ip.id is around 30k then my possible fake RST, ACK packet shows an ip.id filed of 55,502. Looking at Capture PBX B it's ip.id field increments roughly at the same rate as my capture from A.

I believe the RST must be coming from the Cisco 3850 but I'm not sure why or how to prove this yet.

cpocket ( 2019-05-20 13:18:50 +0000 )edit

I should not see a RST packet with RST, ACK in the info field

Yes you would, the ACK just means that it has calculated the value for the ACK number. The ACK does not mean it is ACKing a RST packet. Just that it is ACKing some previously received data.

I believe the RST must be coming from the Cisco 3850 but I'm not sure why or how to prove this yet.

If PBX-A is directly connected to the Cisco 3850, then the RST/ACK is probably not coming from the this device, as it most likely would have a IP TTL of 255 instead of 254. Is the Cisco 3850 acting as a L3 hop? If so, then the RST/ACK packet is most likely sent by the next L3 device to which the Cisco 3850 is connected.

Can you capture on the upstream side of the ...(more)

SYN-bit ( 2019-05-20 13:51:50 +0000 )edit

Hello,

Yes, the 3850 which the PBX is connected to does have routing turned on. I have captured at all 4 paths so I do have that upstream capture. My RST/ACK packet does not appear in any of those other captures.On the capture with the RST/ACK packet the identification field is 55502 and that same number on the upstream router shows me it's a TCP Keep alive from pbx A to B.

cpocket ( 2019-05-20 16:01:06 +0000 )edit

Just curious, are you able to share those 4 capture files somewhere on a file-sharing service? Or is there sensitive data involved?

SYN-bit ( 2019-05-20 17:57:06 +0000 )edit

Hello, I cannot. However, I did write up the following as I'm hoping you could answer my questions within there. This is looking at 8 packets between A and B and just using the capture from A. Using just the capture from A where I see the TCP reset I believe the network is not the cause. I don't see any retransmission's and round trip time is fine other than when the PBX B is waiting on A to send it something.

(Packet#99) PBX A sends a packet with length 0 over to B that means the next Seq. number from A's PBX will have a starting Seq. number of 3763847337. A's PBX also has an acknowledgment number of 2555008053 which is packet 98's starting SEQ # plus the length of the packet so both PBX's are good up to packet 99.

9 ...(more)

cpocket ( 2019-05-21 14:42:06 +0000 )edit

add a comment

TCP RST & RST, ACK question

Comments

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

TCP RST & RST, ACK question edit

Comments

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

TCP RST & RST, ACK question