Ask Your Question
0

Seeing multiple responses but only one request

asked 2022-04-26 18:18:03 +0000

vnonymous gravatar image

updated 2022-04-27 09:14:53 +0000

grahamb gravatar image

Hello,

If there is a better place to ask this question, I apologize. I'm experiencing a very concerning problem that I'm hoping someone more knowledgeable in web protocols may be able to help with. Unfortunately, I cannot share complete .pcapng files because they contain sensitive information. I tried to sanitize them with TraceWrangler, but I think because I needed to decrypt the 802.11 packets with an SSID and password to even view them in Wireshark, TraceWrangler is unable to modify them. Anyway, I digress...

The general scenario is that we have devices in the field that call an endpoint periodically, and by periodically I mean every time they want to connect to any of our services. Recently, we began seeing a spike in failures/retries coming from our devices using this service, and upon further investigation, we've narrowed down the problem to a non-compliant http request. The request the device was making contained a trailing CRLF in its headers. Removing that stops the unexpected behavior from ever occurring.

So, problem solved, right? Well, this specific application hasn't changed in years, and because this is new behavior, management wants a more detailed explanation of the problem, specifically the intermittent nature of it. I do not understand enough to give that to them, this is where I am looking for some help.

Here is what I can provide of the Wireshark capture:

On the left side, you can see that the response contains the expected 200 and payload, but another 400 response is appended directly to the end of the response.

Fortunately, the device will instantly retry in this scenario, and continue to do so until it receives a response it understands.

On the right, which begins before the last stream ends, the device retries again and the service returns a 200 without the appended 400.

I've exported both of these requests and diff checked them, I see almost nothing different between them worth noting. The one thing I did find is that in the request that results in the double response, Wireshark shows it is Http Request 1/2. However, the 2nd part of the request doesn't appear in Wireshark:

The request which gets a response without the trailing 400 is shown as [Http Request 1/1] in Wireshark. The size of these requests is identical, so it doesn't appear at first glance that anything at all is different between them.

I'm wondering now if there's anything useful to be gleaned out of the TCP traffic during these streams. My understanding of TCP behavior is surface-level at best. I've done some reading very recently, but maybe someone here may be able to expediate my understanding of the information I need to know. Is it possible there is any answer to this intermittent behavior within these TCP exchanges?

Thank you in advance.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2022-04-26 20:37:44 +0000

SYN-bit gravatar image

The reason Wireshark does not show [Http Request 1/2] in stream 1 is that the second response was not captured, so wireshark did not get to see it. The client did get to see it (as can be seen from the ACK numbers acking both responses).

As for finding a more detailed explanation, one would need more than 1 example of the good and bad behavior to start seeing a pattern. From the limited data provided, I assume the extra (non-compliant) CRLF in the request always triggers the extra HTTP 400 responses. So the difference in the two examples given are the retransmissions of the first response after sending the second response. But further analysis of multiple occurrences is needed to confirm if this is of importance.

What could have changed is the network stability (you mentioned WiFi) at the client side, triggering retransmissions that were not there before. Then together with the non-compliant response to the non-compliant request might trigger a bug in the software on the client. But that is a lot of assumptions and no proving (until at least more occurrences have been analysis or even reproduction of the error can be done).

edit flag offensive delete link more

Comments

If I were to remove the stream filter, you would see after stream 1 the device continues expected execution of its behavior; meaning that it understood the single response properly - ergo no trailing 400. I assume you're talking about [TCP ACKed unseen segment] frames? Thank you for your insight.

vnonymous gravatar imagevnonymous ( 2022-04-26 20:46:39 +0000 )edit

[TCP ACKed unseen segment] means wireshark sees an ACK for data it has not seen. So in this case, the client does see that data (the second response), but wireshark did not (as somehow, it was not captured), so that's why it reports [Http Request 1/1] instead of [Http Request 1/2]

SYN-bit gravatar imageSYN-bit ( 2022-04-26 21:07:49 +0000 )edit

Okay, yes that makes sense. What's strange is that in this specific example, the device appears to have been able to parse the response normally. It continued on making requests to different services, meaning it was able to break out of this retry loop.

vnonymous gravatar imagevnonymous ( 2022-04-26 21:12:16 +0000 )edit

So analyzing a few more examples of where it did not successfully process the response might lead to a pattern :-)

I understand your difficulties in anonymizing the file and thus not being able to share it publicly. Are you able to share it privately (see my user profile)?

SYN-bit gravatar imageSYN-bit ( 2022-04-26 21:17:14 +0000 )edit

All I have at the moment is one more failed attempt, it does appear to be quite different though.

vnonymous gravatar imagevnonymous ( 2022-04-26 21:30:40 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2022-04-26 18:18:03 +0000

Seen: 139 times

Last updated: Apr 27