A few months back I wrote an article about publishing WSUS with a different URL than the FQDN of the server (link). That was the case with a domain joined server where you can fix things with a SPN. But what about a stand-alone WSUS server with a valid SSL certificate issued with a FQDN of the server?
Right out of the box, it won’t work. Once you enable SSL mode on the WSUS, you are not able to open the WSUS MMC Console. Actually, the WSUS services are all running just fine, but the Windows Server Update Services MMC Console won’t let you in.
Well, that’s because the SSL handshake fails with the name check. Your certificate is for the FQDN of the server, but the server itself don’t know what its’ name actually is. What you need to do is configure the Primary DNS Suffix to match the name of the server’s FQDN (the same as on the SSL certificate). Windows, by default, doesn’t do this for you, but instead leaves it empty.
So, if you have a server named server1 and your domain name is company.com. Go to Control Panel -> System and Security -> System and under Computer Name, Domain and Workgroup Settings, click Change Settings. Then one more time Change and then More. There you can set the Primary DNS Suffix for your server. Int this case, type company.com. The change requires a reboot and after that, WSUS MMC Console should open without problems.
Unfortunately, there doesn’t seem to be a switch for netsh or a straightforward PowerShell Cmdlet for doing this.