v9.4.0.0p1 and later enforce permissions on the `logs` folder, leading to undiagnosable crashes of the service after Windows Update

[seansaleh]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

• Have OpenSSH working as a service
• Update OpenSSH via Windows Updates from something older than v9.4.0.0p1 to something newer
• Have the service fail to start because you had somehow created a logs folder at C:\ProgramData\ssh\logs without extremely locked down permissions (I'm unable to confirm this right now, but it could have even been created by running sshd.exe as a user, not as a service)

Expected behavior

• Service fails to start with a log message anywhere at all saying why
• Or service starts anyways
• Or this requirement for the logs folder to also have locked down permissions is documented somewhere, at least at https://github.com/PowerShell/Win32-OpenSSH/wiki/Security-protection-of-various-files-in-Win32-OpenSSH
  • To be fair I figured this out because of the patch notes for v9.4.0.0p1 say

    the sshd service will check the $env:ProgramData\ssh folder permissions upon startup to ensure only SYSTEM and Administrator accounts have write access to the folder; similar to the existing check upon install in contrib\win32\openssh\install-sshd.ps1.

  • And because of the PR add check for prog data folder permissions during sshd service startup openssh-portable#686

Actual behavior

• Service fails to start with error 1067: The process terminated unexpectedly
• No logs show up in Event Viewer indicating why
• Troubleshooting as per https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps does not show any issues, sshd runs fine with psexec -s sshd.exe -d
• It only gets fixed by either running the full install-ssh.ps1 or by doing rm C:\ProgramData\ssh\logs (and thus depending on the parent folder to have proper permissions)

Solutions

• Validate that the issue is undiagnosable by running psexec -s sshd.exe -d successfully
• Run the full install-ssh.ps1
  or
• Run rm C:\ProgramData\ssh\logs (and thus depending on the parent folder to have proper permissions)

Error details

No response

Environment data

$PSVersionTable 

Name                           Value
----                           -----
PSVersion                      7.3.5
PSEdition                      Core
GitCommitId                    7.3.5
OS                             Microsoft Windows 10.0.20348
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Version

OpenSSH_for_Windows_9.5p1

Visuals

No response

[seansaleh]

To be clear why I believe this is a notable issue:

I installed OpenSSH from the Windows Feature on a number of machines.
I was able to SSH to my servers just fine.
I updated Windows Server on Patch Tuesday.
OpenSSH got updated.
I was no longer to access or manage my servers, because the OpenSSH service would no longer start.
I was not able to diagnose the issue for a long time because there were no errors in the event viewer, and running sshd.exe as per the docs for troubleshooting didn't identify the issue, it worked just fine!
I had to physically connect to each machine individually to fix the OpenSSH folder permissions settings on the log folder.

Generally speaking I'd expect that using Windows Update would be safe to update my servers with, and I'd still maintain access to them after running Windows Update.

[Miraclys]

Thank you, I've wasted the morning looking for mistakes!

[fmiermont]

Same issue here, after the patch tuesday OpenSSH server won't start, and nothing in the log even if set in DEBUG level.

The Log folder was previously created by OpenSSH... Just a to delete it, start the server : working ok.

[crondrift]

Doesn't work for me. Installed it via msi, stopped the service, deleted the folder C:\ProgramData\ssh\logs, started the service. The Log-Folder then gets recreated, but the service keeps crashing. Could someone who got it working post the exact folder permissions please?

[Superberti]

I also tried 9.8.0.0preview and I used the install-ssh.ps1 script.
Unfortunately the sshd still doesn't want to start.
Firewall rule created for ssh with

"New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH SSH Server' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22 -Program "C:\OpenSSH-Win64\sshd.exe""

and even tried to give sshd an firewall exception for everything.

From an ELEVATED power shell with ./sshd -ddd I got:

debug1: get_passwd: lookup_sid() failed: 1332.
debug1: rexec_argv[1]='-ddd'
debug3: using c:\openssh-win64/sshd-session.exe for re-exec
debug2: fd 7 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 7 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
debug3: socketio_bind - ERROR:10013
Bind to port 22 on :: failed: Permission denied.
debug2: fd 7 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
debug3: socketio_bind - ERROR:10013
Bind to port 22 on 0.0.0.0 failed: Permission denied.
Cannot bind any address.

Windows server 2022 21H2, Developer mode ON
Any idea where to look else? This doesn't look too bad, but I wonder why port 22 is still not accessable...

[seansaleh]

@crondrift and @Superberti I think your issues are unrelated (see #2281 for issues with 9.8). This issue is specifically about existing working installs when the service was working before and update, but failing afterwards and when the troubleshooting steps (specifically running psexec -s sshd.exe -d) don't show any issues. I'm also going to update the issue to make that clearer.

(@crondrift make sure you've tried both https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps and powershell.exe -ExecutionPolicy Bypass -File install-sshd.ps1)

[giacomoscoccia]

Same Issue here, same morning wasted.

Deleting logs folder seems solve the issue, but...
DO NOT NAVIGATE the logs folder for any reason. It seems that Windows adds additional permissions that will prevent the service from starting again.

In fact, Windows notices that the folder cannot be accessed and prompts for authorization to permanently gain access to the folder.

Evidently, pressing the Continue button adds additional permissions, which will then cause the sshd.exe service to fail to start.

[twhug]

Thanks for reporting this and for providing the solution. Pretty appalled that this was released without a proper heads-up. Lost a good hour or two this morning chasing a fix until I saw yours.

[crondrift]

@seansaleh
Sorry, my fault. I forgot to mention, that I've updated from v9.5.0.0p1-Beta and everything was working fine until the update to v9.8.0.0p1-Preview. Already tried the troubleshooting and the install-sshd.ps1.

[tgauth]

Same Issue here, same morning wasted.

Deleting logs folder seems solve the issue, but... DO NOT NAVIGATE the logs folder for any reason. It seems that Windows adds additional permissions that will prevent the service from starting again.

In fact, Windows notices that the folder cannot be accessed and prompts for authorization to permanently gain access to the folder.

Evidently, pressing the Continue button adds additional permissions, which will then cause the sshd.exe service to fail to start.

Sorry for all the headaches around this, and thank you for raising the issue here. To be clear, read access to the logs by any user is still permitted, but using the prompt for access grants full control. If you press 'continue` on the prompt, you'll just need to manually restrict the permissions in the folder properties to read/read & execute/list folder contents.

[giacomoscoccia]

Sorry for all the headaches around this, and thank you for raising the issue here. To be clear, read access to the logs by any >user is still permitted, but using the prompt for access grants full control. If you press 'continue` on the prompt, you'll just >need to manually restrict the permissions in the folder properties to read/read & execute/list folder contents.

Thanks por clarification.
Why not consider creating the logs folder with at least read permissions for the administrators group?

[user8446]

Is anyone else seeing connections and disconnects not logged anymore in v9.8?

EDIT: Opened separate issue: #2292

[tgauth]

Why not consider creating the logs folder with at least read permissions for the administrators group?

Full control to the logs folder is permitted for the administrators group and the SYSTEM