FreeBSD 10 changed the supported Xen configurations, notably providing Xen support by default in GENERIC using PVHVM mode. Previously, I’d been using FreeBSD 9 (i386) using a XEN kernel and PV mode, which is deprecated in FreeBSD 10. (If you need a refresher on Xen modes, I highly recommend the article Xen Modes: What Color Is Your Xen.)

There is a fairly established procedure that I tend to follow when setting up FreeBSD PV domUs, but it was difficult to find a guide on setting up PVHVM domUs on my current stack of open-source Xen (4.1) on Debian old-stable (7) using the xm toolchain. Well, here it is.

ISO mangling

Since I manage my Xen stuff remotely (and I didn’t want to add a dependency on VNC), I decided to have my FreeBSD domU use serial for the xm console. The FreeBSD installer iso doesn’t support headless/serial install by default, but there are instructions for modifying the iso to boot into a serial console. Should that page become unavailable, the gist is that you extract the iso, append console="comconsole" to the iso’s /boot/loader.conf, and then recreate the iso.

Configuring the domU

I’ll assume you already have Xen set up, and have created some sort of disk for your domU to live in. Since I’m using LVM, I created a logical volume for the new domU (which I named “eagle”) with sudo lvcreate -L 10G -n eagle-disk solid-state. To clarify, the new LV is called eagle-disk, and I created it on the solid-state volume group.

Here’s the config file I used, which I put in /etc/xen/eagle.cfg:

memory = 1024
name = "eagle"

# PVHVM stuff
builder = "hvm"
kernel = "hvmloader"
boot = "dc"

vif = [ 'mac=00:16:3E:01:AB:23' ]
disk = [ 'phy:/dev/solid-state/eagle-disk,hda,w',
         'file:/virt/iso/FreeBSD-10.1-RELEASE-amd64-headless.iso,hdb:cdrom,r'
       ]

# Necessary for getting the serial console in `xm console`
serial = "pty"

Note that /virt/iso/FreeBSD-10.1-RELEASE-amd64-headless.iso is where I put my mangled iso. Modify this (and the other options) according to your setup.

Installing

Once your configuration is all set, boot the domU (I did sudo xm create -c /etc/xen/eagle.cfg).

After the pretty bootloader, you’ll probably be greeted with the mountroot prompt, which can’t figure out what to boot from. Use the ? command to enumerate your options — since you’re installing, you want to boot from your mangled iso. When I recreated the iso, one of my mkisofs parameters was -V Headless_install, so the mountroot prompt presented my iso option as iso9660/Headless_install. Thanks to my config file, the iso also appeared as cd0. Whichever one you use, tell mountroot to boot from cd9660:name-of-your-device.

You’ll also get a prompt for your system’s console type — I chose the default (vt100).

After that, install FreeBSD as normal. The page up and page down keys probably won’t work as expected, but the arrow keys should. I recommend giving yourself a non-root user (with the wheel group) when you get the option, and enabling the ssh daemon: in case the serial console gets weird on reboot (like it did for me), ssh’ing in as a regular user is much easier than using ed to set PermitRootLogin yes in /etc/ssh/sshd_config and then doing service sshd onerestart.

At the very end, before rebooting, take the option to open a shell in the newly installed system. The iso knows to use a serial console, but your new install doesn’t, so you’ll need to put console="comconsole" in /boot/loader.conf again.

If you reboot right now, your domU will still want to boot from the iso. This is because of the boot = "dc" line in the Xen configuration file — the letters are Windows-style disk letters indicating that the second “disk” (hdb, the iso) has boot priority over the first (hda). Change this line to just boot = "c", remove the iso’s disk entry, and destroy (shutdown) and re-create the domU.

If you get a mountroot prompt again on reboot looking for the install cd, you’re probably still booting from the iso.

Lo and behold

At this point, you should be able to treat your shiny new domU like any other. Have fun!