Ask Your Question

What would cause a RST,ACK from client to SMB server?

asked 2020-04-18 07:29:22 +0000

galapogos gravatar image

updated 2020-04-18 14:28:46 +0000

Hi, I'm trying to troubleshoot a problem I have with a Windows PC connecting to an Synology DS218J NAS on SMB2. I keep getting errors whether connecting via hostname or IP address directly, even when Windows Defender firewall is disabled. The NAS server is working fine as I can access its web portal from the same PC, and I can also access the SMB file shares with other mobile devices/laptops. I've tried looking at the SMB version settings (SMB2 is enabled on the NAS), and even did a Network Reset on the PC, but still no go, so I decided to delve deeper by looking at a packet capture.

What I see when I try to connect to the NAS on Windows Explorer, is the following:

  1. TCP 3-way handshake ([SYN],[SYN,ACK],[ACK])
  2. Negotiate Protocol Request from PC to NAS
  3. TCP ACK from NAS to PC
  4. Negotiate Protocol Response from NAS to PC
  5. [RST,ACK] from PC to NAS

Steps 1-5 will then repeat twice (total 3x) before ultimately failing.

What could be wrong here? Is it a TCP problem, an SMB problem or some other problem?

Attached screenshot of the negotiate protocol response from the server. image description

edit retag flag offensive close merge delete


A (reasonable) guess would be that the PC doesn't like the Negotiate Protocol Response. Sharing a capture with that packet would be very helpful. What OS is the PC running?

grahamb gravatar imagegrahamb ( 2020-04-18 08:31:22 +0000 )edit

Here it is in the original post. Let me know if you want to expand anything. Not sure what I should be looking at.

galapogos gravatar imagegalapogos ( 2020-04-18 14:31:02 +0000 )edit

You've shown us a screenshot of the RST, ACK which is the client closing the connection. What we really need to see id the SMB Negotiation in a format that allows inspection, i.e. a Wireshark capture.

Please share the capture using a public share such as Goggle Drive, DropBox etc. and post a link to it back here.

grahamb gravatar imagegrahamb ( 2020-04-18 15:12:48 +0000 )edit

Hi, here's the link I notice from comparing a working and non working capture, that the working one has some NTLM information in the Negotiate protocol request, while the non working one (which is the link above) doesn't have. Not sure of the significance.

galapogos gravatar imagegalapogos ( 2020-04-18 15:19:57 +0000 )edit

I think you'll have to wait for an SMB expert to take a look. The response looks OK to me, albeit only version SMB 2.0.2, so an older version, but the client does say that's supported in its request.

grahamb gravatar imagegrahamb ( 2020-04-18 16:04:56 +0000 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2020-04-19 09:15:01 +0000

Eddi gravatar image

Root Cause: Security Settings

This is most certainly because the Synology NAS only supports NTLM authentication and the workstation refuses to use NTLM. This behavior is configured through the Windows security policies editor.

How to fix it

Open a command line (cmd.exe) with administrative rights. Then type in secpol.msc, hit enter. This is one of many methods to launch the security policies editor.

Navigate to Security Settings -> Local Policies -> Security Options. The right side will show a number of security options. Locate the option Network security: Restrict NTLM: Outgoing NTLM traffic to remote servers. If this option is marked as "deny all" the trace will look exactly as the one that you have provided.

Dirty details

Part 1: Negotiate Protocol Request Your workstation offers the latest and greatest in SMB dialects, up to and including SMB 3.1.1. It also promises to use all the bells and whistles offered by SMB3, if only the server would play along: Large MTU, directory leasing, encryption, compression.

These details are specified in the field smb2.capabilities and the Negotiate Contexts provided in the request.

Part 2: Negotiate Protocol Response The server selects SMB 2.0.2, which is the oldest version of SMB 2, going back to Windows Vista / Server 2008. Remarkably the server supports signing smb1.sec_mode.sign_enabled is set to 1, but does not insist (required = 0). As this matches the clients request it will not lead to a broken connection.

The interesting part comes in the Security Blob provided by the server. Note that the server only offers one method for authentication: NTLMSSP. A domain joined server would also offer Kerberos.

Since the workstation refuses to use NTLM but insist on "something else" the handshake cannot continue.


The default NTLM settings would allow the connection. Since this is denied in your case someone has changed the settings.

If your workstation is a domain member this was most likely configured by your domain administrator. You cannot override the settings chosen by your domain admin. In this case you (or more likely your domain administrator) need to change the configuration using the group policy editor.

Or (if your Synology firmware supports the configuration) you can join the NAS to the domain.

Good luck


edit flag offensive delete link more


OMG you are a lifesaver. Changing this setting to "Allow all" immediately solved the problem! I don't remember ever changing this though.

However, are there any security implications to allowing all?

galapogos gravatar imagegalapogos ( 2020-04-19 09:42:26 +0000 )edit

For a home network not really.

In an enterprise network I would stick to Kerberos only - that is, if none of the users is bringing own their own server.

Eddi gravatar imageEddi ( 2020-04-19 10:00:54 +0000 )edit

Yup it's a home network. Thanks!

galapogos gravatar imagegalapogos ( 2020-04-19 10:10:50 +0000 )edit

So it was auth, no option but to use NTLM for a non-domain scenario. Odd that the Synology still runs such an old SMB version though. Can it be upgraded?

grahamb gravatar imagegrahamb ( 2020-04-19 19:58:59 +0000 )edit

I can only choose Smb2, or Smb2 with large MTU support, as the minimum SMB version. I've increased my max to SMB3 though.

galapogos gravatar imagegalapogos ( 2020-04-20 02:28:50 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2020-04-18 07:29:22 +0000

Seen: 7,719 times

Last updated: Apr 19 '20