Slow transfer - not so trival

edit

Hello,

It's the first time I'm looking for help on pcap so I hope someone will be able to add something to it.

I have 4x servers in two locations. When sending between two servers in same location - all is good. When sending between locations it's slow. So it's not the servers issue as in the same locations they are working correctly.

I'm only able to capture traffic on servers - source and destination, no middle device. I should be seeing speed around 500Mb but I see 15Mb.

Countless graphs and checking later I see similar pattern in working and not working : At the start both of them are sending a lot of data and having some problems later. https://imgur.com/a/94I2lwL

However the difference is later. On the slow one what I see on the tcpgraph when looking from the destination side : https://imgur.com/nV8MzkH Why so many ACK and why does it take so long ?

My RTT is very low, below 1ms . The data starts fast but later gets slow. I understand dup ack, retransmissions and lower congestion window. However here both of the connection hit hard start with re transmitting a lot of packets but it looks like the slow one is suffering something additionally that after a hard start is goes slowly while the fast one goes smoothly. https://imgur.com/2D5hmdz

I see a lot of out-of-order and later re-transmissions of them with some dup-ack with no re-transmissions. So don't know if this indicates some buffer somewhere ? TCP window size is big, no problems here, not hitting it. No sequence number randomization, bytes in flight stable. https://imgur.com/3WwxJ9J https://imgur.com/f6ESAnx https://imgur.com/8u6cL7g

Don't know where to look or which device to blame and why to blame it. I looked at some sharkfest pretensions but most of them are showing easier problems and here is the first time I'm stuck so any advise would be much appreciated .