Help with network connection issues - TCP Dup ACK / Retransmission

asked 2023-12-13 14:16:28 +0000

JST5000
1 ●2

updated 2023-12-13 16:50:11 +0000

Hello, I have been trying to investigate an issue and was hoping for some help/guidance as I am only semi-familiar with using WireShark and analyzing network captures.

Summary of issue:

System A makes API calls to System B on a regular basis.
All of a sudden, these calls started to fail (all were failing). After roughly 36 hours, the calls started working again.
During this time that the calls were failing we saw the following:
- If we sent an authenticated request to the API, it responded just fine with an unauthorized.
- Even if we sent an empty body with an authorized request, the system responded with an error (invalid request).
- It was only if we sent a real request with data that we would receive an error: "The underlying connection was closed: An unexpected error has occurred"
- We also noticed that if we tested the connection from a different machine (System C) during this time, that the API call worked correctly (no issue).

Based on the information above, we are thinking that the issue is somehow related to the machine mentioned as System A above. We were able to run a network capture and at the end I see a series of TCP DUP ACK and Retransmissions. I have tried to look into this, and have found mention of it possibly being packet loss or it could be fast retransmission. But I was hoping that someone with more expertise in this area could take a look and weigh in with their thoughts.

I was trying to attach the capture or an image, but I don't have enough points yet. So I added the image to this site instead: https://ibb.co/0r4DpMR

System A (our system) - shown as 192.168.0.1 in the capture image.

System B (client system) - shown as 10.10.10.1 in the capture image.

Thanks!

edit retag flag offensive close merge delete

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2023-12-26 23:16:53 +0000

Christian_R

2059 ●11 ●73 ●51 http://crnetpackets.com

Hello, you always have a loss of packets when the packets are too big. In your trace 1418 bytes (Layer4) is still good 1460 (Layer4) = (1500 Layer1) is bad. So it seems you are tunneling the packets. So either you allow bigger packets in the tunnel, reduce MTU at the endpoints, or use some packet manipulation like adjust mss on the path.

BR Christian

edit flag offensive delete link

add a comment

answered 2023-12-13 16:15:29 +0000

hugo.vanderkooij
76 ●190 ●4

Sounds to me you need to look for trouble in System B. If that one throws back errors about unauthenticated connections and all.

But for a proper analysis one needs to capture at least at 2 spots. As close as possible towards the client and similarly as close as possible to the server.

But you picture shows packet loss. Which is detectable by the Selective Acknowledgement packets. Not all server packets get back to the client or at least not in time.

Doing the capture on both ends will give some insigth were packets are lost. But in this case it could very well be in the connection between the two networks and not in the machines themselves.

edit flag offensive delete link

Comments

Thanks for the feedback - very much appreciated. I did want to clarify the systems from the picture, as I left that out of my original description (I'll edit that as well).

System A (our system) - shown as 192.168.0.1 in the capture image. System B (client system) - shown as 10.10.10.1 in the capture image.

I also did want to clarify that the error and unauthorized responses were expected. We were purposely sending bad requests and unauthenticated requests to further test out the network connection. What was surprising to me though, is that these calls seemed to complete just fine from a TCP perspective (connection wasn't closed, etc.). It was only a fully proper request, which I assume meant more data (packets) being sent, which resulted in the connection being closed.

JST5000 ( 2023-12-13 16:49:42 +0000 )edit

add a comment