TCP FIN with Data causing RST

edit

Thanks for the responses. Yes, I thought some device in the middle might be sending the reset but that doesn't seem to be the case. The timing is right, the TTL is right and the IP ID is consecutive. The entire close I described happens with 1 RTT plus 1ms for the client sends. I am capturing right at the edge of our network so the reset are not coming from our gear.

Of course the cloud provider, the ISP, and even us are all saying the same thing, that is, we have thousands of computers and no one is experiencing this problem so its not us. Well, at least we can all agree on that.

Just to verify, does this session closure cause problems? As all data does get acked. The only thing that does not get acked is the FIN from the client.

Also, is this a TLS based protocol? If so, are the last segments of the connection "Application data" or "(Encrypted) Alerts"?

How often do you see these RST packets? Every stream? Or just once in a while?

Since this is a cloud service provider, is their service publicly reachable?

Yes, the reset from the server is logged as a Socket Error by the client application. The client app which is sending email to the cloud service for scrubbing marks the emails as not sent. Result is email stops flowing and the send Q grows. About 6 out of 10 connections end in a reset so some email does get through. Workaround is a script that keeps restarting the send Q with the effect of trying to send more frequently, which in turn means more emails get through. This keeps the Q to a reasonable level.

Yes, this smtp inside ssl. The last FIN with payload is not parsed as an SSL Alert so presumably is app data.

Yes, a publicly accessible service. That has me thinking extreme hardening on the cloud service side and this may be a by product of that.

Interesting, so THEY are sending data in the FIN to you and only when THEY do than, the session is not properly closed. In a session that is normally closed, does the last data and the FIN from their side come right after another or does the FIN from their side come after the last data from YOUR side? If so, than it looks like something on their side loses it's session straight after the (first) FIN was seen.

If it is a publicly accessible cloud provider, can I test it myself? If you don't want to mention their servername here, you could send it by email (my address is in my profile).

Yes, the FIN THEY send is right after a data packet (< 1 ms). This packet payload is always 135 bytes. For a connection that is reset the data in the FIN packet is always 165 bytes. That's interesting, and there are other interesting observations like we have one server that doesn't have this problem, and that the problem comes and goes at times outside any of our changes. All interesting but that still is enough to convince the cloud service provider to get a packet capture in their side.