Running Windows Under iohyve

#How I Learned to Stop Worrying and Love The Windows Operating System Family.

This is a mirror of the maintainers blog

As I wrote about in the past, Windows support was coming to the FreeBSD hypervisor bhyve. Support officially came with FreeBSD 11.0-CURRENT r288524 when it was announced via the FreeBSD Virtualization mailing list. Shortly after, Windows Diplomat Michael Dexter wrote a great how-to on the FreeBSD Wiki to get Windows up and going under bhyve.

The secret sauce to getting Windows running under bhyve is the new UEFI support. This is pretty great news, because when you utilize UEFI in bhyve, you don't have to load the operating system in bhyveload or grub-bhyve first. Here's an example of running a Windows bhyve instance at the command line:

bhyve \
      -c 2 \
      -s 0,hostbridge \
      -s 3,ahci-hd,windows2016.img \
      -s 4,ahci-cd,install.iso \
      -s 10,virtio-net,tap0 \
      -s 31,lpc \
      -l com1,/dev/nmdm0A \
      -l com2,/dev/nmdm1A \
      -l bootrom,BHYVE_UEFI_20151002.fd \
      -m 2G -H -w \
      windows2016

In my spare time, I work on a nifty little shell script called iohyve, so when this support came, I was estatic. I started iohyve because I wanted to move away from Oracle's VirtualBox when doing sandbox testing. Sometimes I like to click on dodgy links and write emails to abuse departments and mark things as spam. I want to put as much separation between me and any potential malware possible. Since most malware is written for Windows users, support in bhyve meant I was once step closer to accomplishing this goal. Now iohyve can't put guests behind a fancy virtual NAT yet, but I did bake UEFI support into iohyve v0.7 which made it to FreeBSD ports recently. For today's tutorial, I am going to be using the GitHub version to accomplish the task of creating a bhyve Windows guest in FreeBSD, since there are less bugs in the iohyve UEFI commands. Please note that since the UEFI Firmware itself is still very new and experimental, iohyve's support of it is also experimental. Right now the limitations are you can only have one virtual HDD, one virtual NIC (using tap devices), and no pass-through. Other operating systems booted either via bhyveload or grub-bhyve can have multiple virtual HDDs, and even passthrough support.

##Host and Installation ISO Preparation.

In order to utilize the bootrom (UEFI) feature in bhyve, you need to be running FreeBSD 11.0-CURRENT at least to revision r288524. You should really be past that point by now if you run 11-CURRENT, though. Since iohyve utilizes ZFS, you should also have a zpool set up. On my laptop, I have choosen to install FreeBSD using Root on ZFS which is fairly simple using the new installer. I'll go into iohyve setup a little later on in the post. Before we even think about running bhyve yet, we need to create a custom Windows installation ISO. Since bhyve doesn't have a video output yet, we must create an "unattended" installation ISO. Since we are going to be remastering a new ISO, we will also tell the unattended ISO to install the VirtIO NIC drivers so the Windows guest can connect to the internet.

As I mentioned before, the FreeBSD wiki entry goes over this in detail. In the time since the Windows support came to bhyve, I wrote a set of small scripts and files in order to try different Windows OS's under bhyve. I found myself trying out different AutoUnattend.xml files over and over again. The XML file is basically a set of "answers" that you would click buttons for when installing Windows to bare-metal. I streamlined that process into those scripts which I called Yabs. I will be using those scripts in this tutorial, but feel free to check out and follow along on the wiki article. You're also going to need to find a copy of a Windows Installation ISO. For this tutorial, I will be using Windows 2008, altough I have tried this with Windows 2012 as well.

-Clone the Yabs repo and start your initial working directory.

$ git clone https://github.com/pr1ntf/YetAnotherBhyveScript.git
Cloning into 'YetAnotherBhyveScript'...
remote: Counting objects: 22, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 22 (delta 9), reused 17 (delta 4), pack-reused 0
Unpacking objects: 100% (22/22), done.
Checking connectivity... done.
$ mv YetAnotherBhyveScript/ win2k8auto
$ cd win2k8auto/

-Fetch firmware from Peter and drivers from The Fedora Project. I will be using the version 0.1-94 as the newer drivers did not work with Windows 2008 for me.

$ fetch https://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
$ fetch https://fedorapeople.org/groups/virt/virtio-win/deprecated-isos/archives/virtio-win-0.1-94/virtio-win-0.1-94.iso

-Since the installation ISO I found is called Win2k8R2.iso my directory looks like this:

$ ls
BHYVE_UEFI_20151002.fd  Win2k8-AutoUnattend.xml extract.sh              remaster.sh             yabs.sh
README.txt              Win2k8R2.iso            null.iso                virtio-win-0.1-94.iso

-We must now edit the extract.sh script to point to the correct locations. Note that running extract.sh requires the FreeBSD Port archivers/pz7ip. Mine looks like this:

#!/bin/sh

# Extract important stuff to remaster folder

folder=win2k8
iso=Win2k8R2.iso
drivers=virtio-win-0.1-94.iso

mkdir -p ${folder}/virtio

7z x ${iso} -o${folder}

tar xf $drivers -C ${folder}/virtio

-After we run ./extract.sh we can copy the AutoUnattend.xml file to our win2k8 directory. The AutoUnattend.xml included with Yabs will set the Administrator password to R3dm0nd! and set the default NIC with an IP of 192.168.0.111 with a specific gatway and subnet. Be sure to go in and edit yours accordingly first. You can remove the second <SynchronousCommand wcm:action="add"> from the file. (The command that runs netsh interface ipv4 set address name="local area connection" source=static address=192.168.0.111 mask=255.255.255.0 gateway=192.168.0.1 on the first login).

$ cat Win2k8-AutoUnattend.xml 
<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="windowsPE">
        <component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" 
language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DiskConfiguration>
                <Disk wcm:action="add">
                    <DiskID>0</DiskID>
                    <WillWipeDisk>true</WillWipeDisk>
                    <CreatePartitions>
                        <CreatePartition wcm:action="add">
                            <Order>1</Order>
                            <Size>400</Size>
                            <Type>EFI</Type>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>2</Order>
                            <Size>128</Size>
                            <Type>MSR</Type>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>3</Order>
                            <Extend>true</Extend>
                            <Type>Primary</Type>
                        </CreatePartition>
                    </CreatePartitions>

                    <ModifyPartitions>
                      <!-- EFI system partition (ESP) -->
                      <ModifyPartition wcm:action="add">
                        <Order>1</Order> 
                        <PartitionID>1</PartitionID> 
                        <Label>System</Label> 
                        <Format>FAT32</Format> 
                      </ModifyPartition>

                      <!-- Windows partition -->
                      <ModifyPartition wcm:action="add">
                        <Order>2</Order> 
                        <PartitionID>3</PartitionID> 
            </WindowsFeatures>
            <Themes>
              <ThemeName>Classic Theme</ThemeName>
              <DefaultThemesOff>true</DefaultThemesOff>
            </Themes>
            <ShowWindowsLive>false</ShowWindowsLive>

            <FirstLogonCommands>
               <SynchronousCommand wcm:action="add">
                  <CommandLine>cmd /C bcdedit /emssettings emsport:1 emsbaudrate:115200</CommandLine>
                  <Description>Enable EMS</Description>
                  <Order>1</Order>
               </SynchronousCommand>
               <SynchronousCommand wcm:action="add">
                    <CommandLine>netsh interface ipv4 set address name="local area connection" source=static 
address=192.168.0.111 mask=255.255.255.0 gateway=192.168.0.1</CommandLine>
                    <Order>2</Order>
                </SynchronousCommand>
            </FirstLogonCommands>
        </component>
    </settings>

    <settings pass="offlineServicing">
      <component name="Microsoft-Windows-PnpCustomizationsNonWinPE" processorArchitecture="amd64" 
publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="NonSxS" 
xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <DriverPaths>
            <PathAndCredentials wcm:action="add" wcm:keyValue="1">
                <Path>d:\virtio</Path>
            </PathAndCredentials>
        </DriverPaths>
      </component>
    </settings>
</unattend>
$ cp Win2k8-AutoUnattend.xml win2k8/AutoUnattend.xml

-Now we edit the remaster.sh script with the correct locations for everything and make our unattended installation ISO. Run ./remaster.sh to wrap everything up in a nice ISO file. It requires the sysutils/cdrtools-devel port. Mine looks like this:

#!/bin/sh

# Remaster new ISO

folder=win2k8
iso=win2k8.iso

mkisofs \
	-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
	-iso-level 4 -J -l -D \
	-N -joliet-long \
	-relaxed-filenames -v \
	-V "Custom" -udf \
	-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
	-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
	-no-emul-boot \
	-o ${iso} ${folder}

##iohyve installation and preparation.

Now that our working directory should look like this:

$ ls
BHYVE_UEFI_20151002.fd  Win2k8R2.iso            remaster.sh             win2k8.iso
README.txt              extract.sh              virtio-win-0.1-94.iso   yabs.sh
Win2k8-AutoUnattend.xml null.iso                win2k8

We can go ahead and install iohyve and get ready for our Windows installation process. As mentioned before, we are going be using the GitHub version, not the version from FreeBSD ports. Running iocage version should output 0.7.1.

$ git clone https://github.com/pr1ntf/iohyve.git
Cloning into 'iohyve'...
remote: Counting objects: 904, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 904 (delta 0), reused 0 (delta 0), pack-reused 901
Receiving objects: 100% (904/904), 231.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (505/505), done.
Checking connectivity... done.
$ cd iohyve/
$ sudo make install clean
Password:
gzip -cn iohyve.8 > iohyve.8.gz
mkdir -p /usr/local/sbin
install -c -m 555 /usr/home/pr1ntf/iohyve/iohyve /usr/local/sbin/
install -c /usr/home/pr1ntf/iohyve/rc.d/* /usr/local/etc/rc.d/
install -c iohyve.8.gz /usr/local/man/man8/
rm -f iohyve.8.gz iohyve.8.cat.gz
$ cd ~
$ iohyve version
iohyve v0.7.1 2015/12/08 Im Here for the Party Edition

Now we need to tell iohyve three things to set it up.

• What zpool to store the iohyve datasets.

• What ethernet interface to attach the hardcoded (for now) bridge0 device that all tap devices will attach to.

• Tell iohyve to load the nmdm and vmm kernel modules required for iohyve operation.

Becuase I want iohyve to setup the network and kernel modules each time my laptop boots, I will split iohyve setup into two sections: Running iohyve setup pool=zroot for the zpool, then editing the host laptop's /etc/rc.conf file and running service iohyve start for the interface and modules.

Because my default ethernet interface is em0 I add the following lines to the host's /etc/rc.conf file:

iohyve_enable="YES"
iohyve_flags="kmod=1 net=em0"

Now I can run iohyve setup pool=zroot and service iohyve start:

$ sudo iohyve setup pool=zroot
Setting up iohyve pool...
$ sudo service iohyve start
Starting iohyve guests...
Loading kernel modules...
Seting up bridge0 on em0...
net.link.tap.up_on_open: 1 -> 1

Before we can create our Windows iohyve guest, we will need three things from our win2k8auto folder from earlier.

• Most importantly, we need the secret sauce. We need the BHYVE_UEFI_20151002.fd UEFI Firmware file that Peter and the folks down at nahanni have provided to the FreeBSD community.

• We need the win2k8.iso unattended Windows Installation ISO we created earlier.

• Due to a quirk in the way the firmware and Windows talk to each other, we will need the null.iso file which is essentially a blank disc.

To accomplish this, we can use the functions iohyve cpiso and iohyve cpfw:

$ ls
BHYVE_UEFI_20151002.fd  Win2k8R2.iso            remaster.sh             win2k8.iso
README.txt              extract.sh              virtio-win-0.1-94.iso   yabs.sh
Win2k8-AutoUnattend.xml null.iso                win2k8
$ sudo iohyve cpiso win2k8.iso
Password:
Copying win2k8.iso from win2k8.iso...
$ sudo iohyve cpiso null.iso
Copying null.iso from null.iso...
$ sudo iohyve cpfw BHYVE_UEFI_20151002.fd
Copying BHYVE_UEFI_20151002.fd from BHYVE_UEFI_20151002.fd...
$ iohyve isolist
Listing ISO's...
null.iso
win2k8.iso
$ iohyve fwlist
Listing Firmware...
BHYVE_UEFI_20151002.fd

##Windows Guest Creation and Preparation.

First we can create a new iohyve guest called win2k8 with a 32G size virtual HDD:

$ sudo iohyve create win2k8 32G
Creating win2k8...

Now we need to set some properties on the guest so that bhyve may properly launch the guest.

$ iohyve fwlist
Listing Firmware...
BHYVE_UEFI_20151002.fd
$ sudo iohyve set win2k8 fw=BHYVE_UEFI_20151002.fd bargs="-H -w" ram=1024M
Setting win2k8 fw=BHYVE_UEFI_20151002.fd...
Setting win2k8 bargs=-H -w...
Setting win2k8 ram=1024M...
$ iohyve getall win2k8
Getting win2k8 props...
description  Tue_Dec__8_10:30:51_MST_2015
fw           BHYVE_UEFI_20151002.fd
ram          1024M
os           default
cpu          1
size         32G
bargs        -H_-w
loader       bhyveload
name         win2k8
boot         0
tap          tap0
persist      1
con          nmdm0
autogrub     \n
install      yes

Now we can start our first installation boot up.

##Windows Installation Process

Windows has three main phases to it's install process:

• First install pass (copying files from installation ISO)

• Second install pass (Windows is actually extracting and installing things)

• Third install pass (Windows runs a "first logon" set of commands that were given in the AutoUnattend.xml file)

Before the first pass I like to open another console and run iohyve console win2k8.

$ sudo iohyve console win2k8
Password:
Starting console on win2k8...
~. to escape console [uses cu(1) for console]
Connected

Now we can start the win2k8 guest via the iohyve uefi function. It should start right away.

$ iohyve isolist
Listing ISO's...
null.iso
win2k8.iso
$ sudo iohyve uefi win2k8 win2k8.iso

Switch over to your other console and after a while you should see the Windows SAC:

Computer is booting, SAC started and initialized.                               
                                                                                
Use the "ch -?" command for information about using channels.                   
Use the "?" command for general help.                                           
                                                                                
                                                                                
SAC>                        
EVENT: The CMD command is now available.                                        
SAC>                                                                            
EVENT:   A new channel has been created.  Use "ch -?" for channel help.         
Channel: SACSetupAct                                                            
SAC>                                                                            
EVENT:   A new channel has been created.  Use "ch -?" for channel help.         
Channel: SACSetupErr                                                            
SAC>

You can press [Esc]+[Tab] then [Return] to switch to the SACSetupAct console that shows what the installation is doing. The Calling WIMApplyImage portion takes quit a bit of time.

:\ProgramData\] doesn't exist; no need to move it before applying image.
2015-12-08 10:51:09, Info                  IBS    MoveOldOSFiles:File/folder [E
:\Recovery\] doesn't exist; no need to move it before applying image.
2015-12-08 10:51:09, Info                  IBS    MoveOldOSFiles:File/folder [E
:\Users\] doesn't exist; no need to move it before applying image.
2015-12-08 10:51:09, Info                  IBS    MoveOldOSFiles:File/folder [E
:\Windows\] doesn't exist; no need to move it before applying image.
2015-12-08 10:51:09, Info       [0x06412c] IBSLIB SetCheckpoint: Checkpoint("Wi
nPEArchiveOldWindowsFoldersStartCheckpoint") in progress...
2015-12-08 10:51:09, Info       [0x06412e] IBSLIB SetCheckpoint: Checkpoint "Wi
nPEArchiveOldWindowsFoldersStartCheckpoint" successfully set.
2015-12-08 10:51:09, Info       [0x06412c] IBSLIB SetCheckpoint: Checkpoint("Wi
nPEArchiveOldWindowsFoldersDoneCheckpoint") in progress...
2015-12-08 10:51:09, Info       [0x06412e] IBSLIB SetCheckpoint: Checkpoint "Wi
nPEArchiveOldWindowsFoldersDoneCheckpoint" successfully set.
2015-12-08 10:51:09, Info       [0x06412c] IBSLIB SetCheckpoint: Checkpoint("Wi
nPEImageApplyReadyCheckpoint") in progress...
2015-12-08 10:51:09, Info       [0x06412e] IBSLIB SetCheckpoint: Checkpoint "Wi
nPEImageApplyReadyCheckpoint" successfully set.
2015-12-08 10:51:09, Info       [0x06009e] IBS    DeployWIMImage:Calling IDepWI
MImageResolved::Apply...
2015-12-08 10:51:10, Info       [0x0606cc] IBS    Calling WIMApplyImage (flags 
= 0x184)...

The installation then proceeds to copy over some driver files to the virtual HDD then you should be left with the following screen. Note that if it seems to be frozen there, the guest may have already finished it's first pass and shutdown.

Name:                  SAC
Description:           Special Adminis
Type:                  VT-UTF8
Channel GUID:          8472e3a1-9ddc-11e5-9c07-806e6f6e6963
Application Type GUID: 63d02270-8aa4-11d5-bccf-806d6172696f

Press <esc><tab> for next channel.
Press <esc><tab>0 to return to the SAC channel.
Use any other key to view this channel.

Over on the first console, check to make sure the guest is shutdown.

$ iohyve list
Guest   VMM?  Running?  rcboot?  Description
win2k8  YES   NO        NO       Tue_Dec__8_10:30:51_MST_2015

From the info above, we can see that the Running? flag is set to NO meaning the guest has shut down. We also need to run iohyve destroy win2k8 to remove it from VMM? or the guest will fail during the second pass.

$ sudo iohyve destroy win2k8
Destroying win2k8...
errno = 37

Now we can run the second pass:

$ sudo iohyve uefi win2k8 null.iso

If you switch back over to the SAC console you can change to the SACSetupAct channel and watch things scroll past the screen. This step is really fast and sometimes you can't catch the SACSetupAct channel so you are left with a screen that looks like this when the guest has shutdown.

Setup is updating registry settings...
EVENT:   A new channel has been created.  Use "ch -?" for channel help.
Channel: SACSetupAct
SAC>
EVENT:   A new channel has been created.  Use "ch -?" for channel help.
Channel: SACSetupErr
SAC>
EVENT: The CMD command is now available.
SAC>
EVENT:   A channel has been closed.
Channel: SACSetupAct
SAC>
EVENT:   A channel has been closed.
Channel: SACSetupErr
SAC>
The SAC will become unavailable soon.  The computer is shutting down.

SAC>

Switch over to your first console and check to make sure it shutdown again and destroy it again. You can then start the third pass and then eventually boot into Windows for the first time.

$ iohyve list
Guest   VMM?  Running?  rcboot?  Description
win2k8  YES   NO        NO       Tue_Dec__8_10:30:51_MST_2015
$ sudo iohyve destroy win2k8
Destroying win2k8...
errno = 37
$ sudo iohyve uefi win2k8 null.iso

The first logon installtion phase does take quite a bit of time to finish, but eventually, you will be given a screen that looks like this:

Computer is booting, SAC started and initialized.                               
                                                                                
Use the "ch -?" command for information about using channels.                   
Use the "?" command for general help.                                           
                                                                                
                                                                                
SAC>                                                                            
EVENT: The CMD command is now available.                                        
SAC>

If I run the i SAC command, I can see the guest was given the IP address I set in the AutoUnattend.xml file earlier. By default, the Windows guest will not ping on the network, but running nmap 192.168.0.111 will show that Windows Remote Desktop is available.

SAC>i                                                                           
Net: 12, Ip=192.168.0.111  Subnet=255.255.255.0  Gateway=192.168.0.1

That's it! You can now use your trusty remote desktop client to connect to the guest's Administrator account with the password you chose in the AutoUnattend.xml file which is R3dm0nd! by default.

##Cool stuff you can do after installation.

Even though the iohyve uefi function is not officially supported yet, you can still use built in iohyve tools to do cool ZFS things to the Windows guest like create a snapshot of the guest before installing updates:

$ sudo iohyve snap win2k8@preupdate
Password:
Taking snapshot win2k8@preupdate
$ iohyve snaplist
win2k8@preupdate

You can also make an independent clone of the guest once your image is made so you can don't have to run the Windows installation process over and over again:

$ sudo iohyve clone win2k8 win2k8-deploy                                                                                               
Cloning win2k8 to win2k8-deploy
$ iohyve list
Guest          VMM?  Running?  rcboot?  Description
win2k8         YES   NO        NO       Tue_Dec__8_10:30:51_MST_2015
win2k8-deploy  NO    NO        NO       Tue_Dec__8_11:34:38_MST_2015