trebuildulation
[tree-bild-yuh-ley-shuh n]
noun
plural noun: trebuildulations
: grievous trouble or severe trial whilst rebuilding or repairing
: made up word
: nod to Star Trek
"the trebuildulations of a server"
What started with warning about disk space, turned into a complete server rebuild which occupied all of my free time this week. I’m writing down some of the issues for the benefit of my future self and others.
Old Windows
When I see a disk space warning, I head into the drive and look for things to delete. In this case, I immediately noticed Windows.old taking up 15GB – this remnant of the upgrade to Server 2012 R2 last year was ripe for removal.
If this were Windows 7, I would run Disk Cleanup but on Server that requires the Desktop Experience feature to be installed. It isn’t by default and can’t be removed once it is. So, I set about trying to remove the folder at the command line.
At this point, it seems I failed to remove the junctions from Windows.old but succeeded in taking ownership, resetting permissions and removing the folder. The folder was gone but all was not well. On restart, the Hyper-V management service wouldn’t start.
I forget the error but I eventually determined that the permissions had been reset on C:\ProgramData\Application Data. Unrestricted by folder permissions, Windows links this folder to itself resulting in C:\ProgramData\Application Data\Application Data\Application Data\Application Data\Application Data... This causes lots of MAX_PATH issues at the very least.
Despite correcting the permissions, I wasn’t quite able to fix everything and Hyper-V continued to fail. A rebuild was in order.
We couldn't create a new partition
Microsoft have refined the Windows Server installation process over the years so that it is now normally relatively painless, even without an unattended install. Boot from a USB key containing the installation media, select some localisation options, enter a licence key and off it goes. Not this time.
My RAID volume was visible to the installer and I could delete, create and format partitions but when I came to the next step in the process:
We couldn't create a new partition or locate an existing one. For more information, see the Setup log files.
This foxed me for a while. I tried DISKPART from the recovery prompt, I tried resetting disks to non-RAID and I tried disabling S.M.A.R.T. in the BIOS. Nothing worked. I did notice mention of removing USB devices and disconnecting other hard drives suggesting there’s some hidden limit on the number of drives or partitions that the installer can handle. I could have removed each of the three data drives one by one to see if that theory had merit but I decided to jump in and remove all three.
Success! A short while later I had a working Windows Server 2012 R2.
Detached Disks in the Storage Pool
I reconnected the three data drives and now came time to see if Storage Spaces would come back online as promised.
The steps were straightforward:
- Set Read-Write Access for the Storage Space
- Attach Virtual Disk for each of the virtual disks
- Online each of the disks
The disks were back online and the data was available. Great!
This was a short-lived success story – the disks were offline after reboot and had to be re-attached. Thankfully, I was not alone and a PowerShell one-liner fixed the issue.
Get-VirtualDisk | Set-VirtualDisk -IsManualAttach $False
Undesirable Desired State Configuration Hang
I thought I’d take the opportunity to switch my imperative PowerShell setup scripts for Desired State Configuration-based goodness. This was a pretty smooth process but there was one gotcha.
DSC takes care of silent MSI installations but EXE packages require the appropriate “/quiet” arguments. The result of missing arguments in my configuration script meant that DSC sat waiting for someone to click an invisible dialog box in the EXE’s installer.
Having fixed my script, I killed the PowerShell process and re-tried to be presented with this:
Cannot invoke the SendConfigurationApply method. The PerformRequiredConfigurationChecks method is in progress and must
return before SendConfigurationApply can be invoked.
The issue even survived a reboot.
Again, I was not alone and some more PowerShell later, my desired state is configured.
I’m pleased to report the server is back to a happy state and all is well in its world.