Wireshark trace - timeout problem

asked 2019-12-12 09:54:04 +0000

bojanv55 gravatar image

I have following setup (as visible in the attached image):

A (java process) -> B (kubernetes ambassador proxy) -> C (java service in kubernetes pod)

Communication is done using HTTPS between A and B, and then ambassador strips HTTPS and continues talking HTTP with C.

The problem I'm experiencing is that sometimes, it happens that HTTP BODY message that is being sent is not transmitted 100% between A and B, but only on B side trace I can see that it stopped for some reason (in trace on A side it is shown as all is sent OK). Then, java process in C (which is waiting for B-proxy to forward all the data) is just waiting and time-outs after 30 seconds.

You can see in the image attached, in A trace it is written that whole BODY is sent, but in trace on the B side, only half of BODY is visible (delivered). I suspect on these TCP Previous segment not captured.

You can also see that after this it just waits for 30 seconds, and times-out.

It happens pretty often in my setup. Does anyone knows what could be a problem?

enter image description here

edit retag flag offensive close merge delete

Comments

Were A->B and B->A anonymized/modified or is there PAT and NAT in the process?

Chuckc gravatar imageChuckc ( 2019-12-12 13:28:16 +0000 )edit

B is in kubernetes. A is accessing it using this ambassador mapping service that is then forwarding it to the ambassador pods (B). Do you think that B should see all the packages in trace (since A says everything sent and confirmed) ? Maybe then the problem is even before B (before ambassador pod in k8s) ?

bojanv55 gravatar imagebojanv55 ( 2019-12-13 15:08:25 +0000 )edit

image description

The timestamps on some of the packets in B to A are out of order. reordercap will sort them by time which sometimes helps Wireshark with analysis.

Chuckc gravatar imageChuckc ( 2019-12-14 05:51:02 +0000 )edit

Tried that just now. Seems more-less the same. Here is updated image with all packets: https://prnt.sc/qap1zd

Is it possible that wireshark dropped packages since it was recorder inside the pod? Maybe there is some other way to capture packages outside of the pod. I guess since on A side it looks like everything was delivered that something like this is possible? Or maybe this is due to ambassador limiting bandwidth or timeouts?

bojanv55 gravatar imagebojanv55 ( 2019-12-14 09:25:07 +0000 )edit

Does the proxy have statistics available to indicate its performance? Queue size? Drops? Buffer usage?

Chuckc gravatar imageChuckc ( 2019-12-14 14:26:17 +0000 )edit
see more comments