2021.02.16 - Hyper-V 2019 in a Workgroup

How I setup Hyper-V 2019 in a Workgroup environment (mostly how to get Windows Server Core to do useful things)

Intro

It's over due but we are finally replacing our mishmash of servers with two matching servers, running on our -48v DC power plant with SSDs. The servers are build-your-own type things from SuperMicro. Personally I really like Lenovo/IBM servers, but it was hard to beat the SuperMicro price. Most of our servers run on Linux but most of my virtualization experiance has been with Microsoft Hyper-V and I've been very happy with it. Microsoft is also kind enough to offer a free verison of Hyper-V that is full featured, just lacking the Windows Server Desktop experiance. Most of this will be to document how to use powershell, Microsoft's prefered command line interface.

I started with the OS already installed, my boss had already built the servers and installed the OS (of course take the fun part away :p ). My Boss also choice a setup of mirrored SSDs for the OS and a RAID 5/parity type setup with 3 SSDs for storage the virtual server's hard disks. As mentioned above I'll be using these servers in a workgroup setup rather than a domain setup (which would simplfy things, here, but add complexity and other servers I don't want). The servers themselves will using the Hyper-V replication funtion to basically backup each other. The normal setup will be each server running about half of the vitual servers, but with available resources for one server to run everything. It's not as clean as a setup supporting live-migration, but considering the only time I'll need this function will be to do updates (late at night in a maintenance window) or in the case of major hardware failure I'm willing to accept the brief downtime durring switch over considering the huge cost savings.

Server Setup

A couple of things before we get to the actual Hyper-V setup. I made these changes later, but really should have done this first. First thing is to setup the remote control. No not remote desktop but the actaul remote control of the server so if I do something like mess up a network config I don't have to get a keyboard and montior plugged into the server to fix it. SuperMicro has a perfectly fine iLO (integrated Lights Out) system, and you don't need to pay them more money to use it. The only strange thing is the default password is printed in super small print on the top of the server. The default username is ADMIN (case matters!). My only complaint is that it doesn't seem to understand the -48volt power supplies. They are listed as good and working but I don't get any tempature or voltage stats from them.

Also my boss choose to use the Intel Raid controller (a kind of software or fake raid) instead of a hardware raid controller. With that in mind I removed the Raid 5 from the Intel Raid controller and set it up with Windows Storage Spaces instead. I think using Windows Storage Spaces will make the drives easier to manage and also make expanding the storage easier later if we need to. We used Kingston SSDs and I'm happy to find that the Kingston SSD health software installs fine on Windows Server Core and can see the health of the drives behind the Raid controller. Check back with this section in 3 years or so and see if I still think that was a good idea.

One other small thing, use the same password on both servers, I think this might be important when setting up Hyper-V replication

Using Powershell

After installing Windows Server Core and booting it up you get two MS-DOS looking windows, one has a simple menu for some basic server config and one is a legit Command Prompt. If you close either window they seem to stay closed. If you are infront of the server (or have iLO setup) you can use ALT+CTRL+DEL to bring up a little UI which will let you open Task Manager. From there you can do File, Run New Task, CMD and open a few command prompts. Run the command sconfig in one of them to get your simple menu back. If you are Remote Desktop-ed to the server you can use the keycombo ALT+CTRL+END to bring up the window to start Task Manager.

Launching Powershell should be the first thing that you do, either by typing powershell in the command prompt or in the Run New Task. Powershell supports auto completion using TAB of file names, commands and options of commands. There are however a ton of commands so you might be hitting TAB for a while if you don't type enough of the command out. Also (as of Server 2019) most of the basic Linux commands are supported like ls, rm, mv, cat and wget, whoami. As far as I can tell they seem to be aliases of the equivalent powershell command, so don't expect the command line options to be the same.

Here's some powershell commands I found the need to reference occasionally:

Invoke-WebRequest - wget is an alias of this. You'll need to use the option -OutFile to tell it where to save whatever you ask it to download
wget https://nick.5i5.org/index.php -OutFile download.php

Expand-Archive - unzip a file

Restart-Computer - reboots the computer, be careful with this one, it's very Linux like (it just does it, no asking if you are sure)

Stop-Computer - powers off the computer, be careful with this one, it's very Linux like (it just does it, no asking if you are sure)

Invoke-Command - runs a powershell command on another computer. I use this when testing to see if I have access to the servers setup correctly
Invoke-Command {whoami} -ComputerName HPV2021A -Credential HPV2021A\Administrator

Get-NetAdapterStatistics - Get ethernet statistics, by default only shows very limited info, use Format-List to show more info
Get-NetAdapterStatistics -Name "SXB1B CPU1 RP2A Port 2" | Format-List -Property "*"

Making Server Core Easier to Use

Microsoft has done a great job giving Powershell the ability to control almost all of Windows, but still being able to use a GUI in windows is handy when I forget a random powershell command or need to use something like device manager. Microsoft has released a package called AppCompatibility that includes some of those tools:

Sadly it does not include the graphical Hyper-V manager, that was only added into version 1903 and the free Hyper-V server 2019 is based on version 1809. To install the command is (from PowerShell of course): Add-WindowsCapability -Online -Name ServerCore.AppCompatibility~~~~0.0.1.0

After that I added the Intel Raid managment tool. After downloading the file from intel for Intel Rapid Storage Technology enterprise (Intel RSTe) Driver for Intel Server Boards and Systems Based on Intel 62X Chipset and extracting it there's a ton of random stuff in there. The driver is in SW_Packages\F6-drivers\VROCF6Drivers_MUP.zip. In that zip file is a dpInst.bat file which installs the needed drivers. It launches several installers, one after another and each one requires a reboot if it installs something so you may have to run the .bat file more than once. Also make sure there are no spaces in the full path to that file, it seems to upset the .bat script.

Once the drivers are installed and the server rebooted in the SW_Packages\CLI\VROC_6.3.0.1031_CLI.zip is a CLI verison of the program. You can run IntelVROCCli.exe -I as a quick test to see if the drivers were installed correctly. There's also a graphical program that can be installed, I don't have notes for installing that. I think the installer was headless and the software just appeared in the C:\Program Files\Intel\Intel(R) Virtual RAID on CPU folder.

Next I installed the Kingston SSD Manager. This is a nice normal installer and works exactly as expected.

The last couple of things to enable are remote management. I'm going to skip over remote-desktop as that was already setup when I started working on these servers and focus on the things I needed to change to manage the servers with RSAT from a workgroup computer. First make sure the server is up-to-date. Then run (in powershell): Enable-PSRemoting
Enable-WSManCredSSP -Role server

That's it. If you get weird errors check and make sure the Network adapter of the server is set to the Private NetworkCategory: Get-NetConnectionProfile
Set-NetConnectionProfile -InterfaceAlias "Onboard LAN 1" -NetworkCategory Private

EDIT: I also found that by default Server Core is set to the balanced power mode. Probably doesn't make a huge differance by I like to change that to High Performance. To do that first run: powercfg.exe /list to see what it's set to. Then, if you need to change it, use: powercfg.exe /s "{whatever the GUID is of High Performance}"

Setting up Hyper-V Remote Management in a Workgroup

Finally, the bit that is the most annoying to get working correctly. I'll be borrowing heavily from Timothy Gruber's post on "REMOTELY MANAGING HYPER-V SERVER IN A WORKGROUP OR NON-DOMAIN" so if you get stuck it might be worth a read. First on the Windows 10 computer you want to use to manage the server open the Control Panel, Programs and Features and Turn Windows features on or off. You'll need to find Hyper-V on that list, expand it and check the Hyper-V Management Tools box. Hit OK and it should install the Hyper-V manager (you may need to restart your computer). You'll also want to install Server Manager which is found in Settings, Apps & Features, Optional Features, Add a feature. All the server managment features will start with RSAT:. I only installed the Server Manager, Remote Access Managment Tools and Group Policy Management Tools.

Now that you've got the programs installed here's the actual setup. On the Windows 10 computer open powershell. Use the Get-NetConnectionProfile command to make sure your ethernet interface is set to Private, and if it's not then change it. Then launch notepad as admin and edit your hosts file. You'll need to add the names of your Hyper-V servers to that file so your computer can resolve their names to IP addresses. Then you'll need to save the server's credentials, in Powershell or Command Prompt run: cmdkey /add:HyperVServerA.example.com /user:HyperVServerA\Administrator /pass "SecurePassword"
cmdkey /add:HyperVServerB.example.com /user:HyperVServerB\Administrator /pass "SecurePassword" For each server. You can also go to the Control Panel, Credential Manager, Windows Credentials to add or edit these.

Next there's a bunch of powershell commands to run. I don't pretend to understand what they are doing so I'd suggest reading Timothy Gruber's post for a better explanation. These commands assume you are adding two Hyper-V servers named HyperVServerA and HyperVServerB and have no other Hyper-V servers that you need to access. These commands may need to be altered if that is not true.

winrm quickconfig Set-Item WSMan:\localhost\Client\TrustedHosts -Value "HyperVServerA.example.com,HyperVServerB.example.com" Enable-WSManCredSSP -Role client -DelegateComputer "HyperVServerA.example.com,HyperVServerB.example.com" New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\" -Name 'CredentialsDelegation' New-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\" -Name 'AllowFreshCredentialsWhenNTLMOnly' -PropertyType DWord -Value "00000001" New-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\" -Name 'ConcatenateDefaults_AllowFreshNTLMOnly' -PropertyType DWord -Value "00000001" New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\" -Name 'AllowFreshCredentialsWhenNTLMOnly' New-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowFreshCredentialsWhenNTLMOnly\" -Name '1' -Value "wsman/HyperVServerA.example.com" New-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowFreshCredentialsWhenNTLMOnly\" -Name '2' -Value "wsman/HyperVServerB.example.com"

I don't know if it's required, but I'd give the computer a restart at this point. Then you should be able to open Server Manager or Hyper-V Manager and see and connect to your servers. At this point I'm going to pause the Hyper-V setup and move onto...

Microsoft Storage Spaces

As mentioned in the intro I'm going to use Microsoft Storage Spaces rather then the Intel Virtual Raid (or fake raid or Intel Rapid Storage Technology enterprise or whatever they are calling it now) as I think in a Raid 5 / Parity setup the Microsoft Storage Spaces will be more robust. If this turned out to be a horrible mistake I'll try to come back and update this section. Now that we can remotely manage the server you can use Server Manager to setup storage spaces. Under the File and Storage Services section go to Volumes and Storage Pools. Hopefully in there you'll see your Hyper-V server listed and it should show Primordial (meaning there are blank unpartitioned hard drives in it).

If you don't see any drives be sure to reset them (Right-Click on the drive and hit reset) and do a "Rescan-Storage" when done (otherwise they won't show up). Right-Click on it to create a new Storage Pool.

EDIT: Please check out this article I just found on the issues of Allocation Unit Size and Microsoft Storage Spaces when using Parity mode:https://wasteofserver.com/storage-spaces-with-parity-very-slow-writes-solved/ Apparently there are some issues created using the default settings. In my case with 3 drives the default interleave is 256k which matches terribly with the NTFS default Allocation Unit Size of 4k. Formatting the drive with a Allocation Unit size of 512k matches better (half of the 512k goes to each drive matching the 256k interleave).

Once the Pool is created (I named my, very creativly, Pool1) you should be able to use the same tool to create the Virtual Disk but for some reason I was never able to get it to work. Instead I used Powershell on the Hyper-V server to create it: New-VirtualDisk -StoragePoolFriendlyName Pool1 -FriendlyName Data -ResiliencySettingName Parity -Size 1.6TB -ProvisioningType Fixed -MediaType SSD -PhysicalDiskRedundancy 1 After running that command refresh the Server Manager - Storage Pools window and you should now see a drive listed under Virtual Disks. Change to the Disks view (on the side above Storage Pools and you'll see your new drive listed as Offline. Right-Click on it and hit Bring Online then once it's online Right-Click on it again and hit New Volume and format the drive as required.

Setting Up Replication - DNS Suffix

For this step I'll generally be following this guide I found on Microsoft's Forums. First the Hyper-V servers will need some way to talk to eachother. I'd suggest a dedicated network connection between them. You don't want to bog down other useful traffic when replicating data. Also then you can put 10G ethernet cards in and replicated over a 10G link without needing a 10G switch. You'll probably also need to add a DNS suffix to the computer name, apparently it makes the certifcate generator mad when there isn't one. To add a DNS suffix I ran: reg.exe add HKLM\SYSTEM\CurrentControlSet\services\Tcpip\Parameters /v "NV Domain" /t REG_SZ /d "example.com" /f
Also, disable windows trying to publish DNS updates: reg.exe add HKLM\SYSTEM\CurrentControlSet\services\Tcpip\Parameters /v "DisableDynamicUpdate" /t REG_DWORD /d 1 /f

I'd suggest using a real domain that you own, even using stuff like .int or .network is tricky as those are real domains that can be registered. .local is an option, but might conflict with other things, so just use a real domain that you own. Reboot the server and then check ipconfig /all and it should now be listed under Primary DNS Suffix.

Now that you've added a domain/DNS suffix you'll need to make a few changes to your windows 10 computer otherwise you'll get managment errors. If you haven't added the server name (like HyperVServerA.example.com) to your DNS you'll need to add that to the hosts file on your windows 10 computer and both Hyper-V servers. You'll also need to add it to the list of trusted hosts on the windows 10 computer: Set-Item WSMan:\localhost\Client\TrustedHosts -Value "HyperVServerA,HyperVServerB,HyperVServerA.example.com,HyperVServerB.example.com"

Setting Up Replication - Certificates

You'll need to generate and install the required certificates to enable replication without a domain. the forum post walks you through how to do this in Windows with makecert.exe but since that program is a pain to get (Microsoft bundles it with other development tools) I'm going to be following these directions on how to do it with PowerShell. The linked guide has you generate certificates that expire after 1 year. I'll be changing that to 15 years. This is probably a bad idea and you should just renew the certificates every year. Just run these commands on one server in powershell:

$date=Get-Date New-SelfSignedCertificate -DnsName "HyperVServerA.example.com" -notafter $date.AddYears(15) -CertStoreLocation "cert:\LocalMachine\My" -TestRoot New-SelfSignedCertificate -DnsName "HyperVServerB.example.com" -notafter $date.AddYears(15) -CertStoreLocation "cert:\LocalMachine\My" -TestRoot

Then you'll need to open the Microsoft Management Console to move the certificates around. I'll be opening this on my windows 10 computer by typing mmc in Run. Then go to File, Add/Remove Snap-ins and add certificates. On the popup choose Computer Account and on the next screen choose the hyper-v server you ran the certificate commands on, then do this again and add the second Hyper-V server. Go to the Personal\Certificates store and you should see the two certs we made. Go to Intermediate Certification Authorities\Certificates and you'll see CertReq Test Root Copy that and paste it in Trusted Root Certification\Certificates on both servers.

For some reason I couldn't export the certificate for HyperVServerB from MMC so here's a handy powershell command to run on the server that generated the certificates to export it: Get-ChildItem -path Cert:\* -Recurse | where {$_.Subject -like '*HyperVServerB*'} | Export-PfxCertificate -FilePath "$home\export.pfx" -Password $(ConvertTo-SecureString -String "1234" -Force -AsPlainText) This will put the export.pfx file in C:\Users\Administrator\, just copy it to the HyperVServerB server, and run this command to import it on HyperVServerB: Import-PfxCertificate -FilePath .\export.pfx -CertStoreLocation cert:\LocalMachine\My -Password $(ConvertTo-SecureString -String "1234" -Force -AsPlainText)

The last thing you'll need to do with certificates is disable the revocation check, as it will always fail on these self signed certs. To do that run this on both servers: reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Replication" /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f

Setting Up Replication - Final Steps

On both Hyper-V servers you'll need to alter the firewall rules to permit the replication traffic. Run this powershell command on each server: Enable-NetFirewallRule -Name VIRT-HVRHTTPSL-In-TCP-NoScope Then add credentials and set them as trusted hosts for eachother, so on HyperVServerA run: cmdkey /add:HyperVServerB.example.com /user:HyperVServerB\Administrator /pass:SecurePassword Set-Item WSMan:\localhost\Client\TrustedHosts -Value "HyperVServerB.example.com" and on HyperVServerB: cmdkey /add:HyperVServerA.example.com /user:HyperVServerA\Administrator /pass:SecurePassword Set-Item WSMan:\localhost\Client\TrustedHosts -Value "HyperVServerA.example.com"

Finally there's just a couple of odds and ends to setup in the Hyper-V manager. If you haven't done so yet you'll want to setup the default switch so your Virtual Machines can reach the outside world. This is where having iLO setup is handy, if you choose the wrong network adaptor you'll lose access to the server. To setup the virtual switch first get a list of the network adaptors on your server with: Get-NetAdapter. To view the IP address assigned to an ethernet adapter use: Get-NetIPAddress -InterfaceAlias "{Name from get-netadapter here}" To actaully create the virtual switch run: New-VMSwitch -Name ExternalSwitch -NetAdapterName "{Name from get-netadapter here}" -AllowManagementOS $false

On the Microsoft Storage Spaces drive I created the following folders (it's the G drive in these servers):

mkdir G:\Hyper-V mkdir G:\Hyper-V\Config mkdir G:\Hyper-V\VirtualHD mkdir G:\Hyper-V\ReplicatedHD

In Hyper-V Manager open the open of the servers and click on Hyper-V Settingsand set the Virtual Hard Disks and Virtual Machines to the directories created above. Then go to the Replication Configuration section and Enable this computer as a Recplica server. Then check Use certificate-based Authentication (HTTPS):. Click Select Certificate and choose (what should be the only option) the correct Cert for the server. Then check Allow replication from any authenticated server and choose the final directory created to store the Replica files.

-Nick