bhyve is a hypervisor for FreeBSD. A few weeks ago I gave it a try and I must admit I was very impressed, so much that I decided to write a short tutorial on how to configure bhyve to boot from NFS filesystem. So lets start.

bhyveload(8) & bhyve(8)
First let’s explain the difference between bhyveload(8) and bhyve(8) programs. bhyeload(8) is used only for creating virtual machines with FreeBSD. In this command you can give several different arguments, for example from with FreeBSD iso it should be booted. bhyve(8) is more general and can be used to run and create virtual machines with other operating systems. What is important after running bhyveload(8) you create VM but it is still not running for this you still need to run bhyve(8) with the same amount of memory and the same VM name.

One thing which troubles me is that bhyve(8) is quite complicated to use, there are so many option that you need to give to it. There is a very nice shell script (/usr/share/examples/bhyve/vmrun.sh) which simplifies work a lot. Unfortunately when I’m writing this tutorial it isn’t prepared for our case.

Last think to mention is that we need to load one bhyve kernel module called vmm.ko, you can do it simply by calling it:
# kldload vmm
NFS export
How to configure a NFS server in FreeBSD, you can read more here.

Creating network card and bridge
Now we need to create a network device for our virtual machine. For that we can use a tap(4) device, which is a software loopback mechanism. To create such a device we need to follow 3 steps:

1. load kernel module:
   # kldload if_tap
2. create device:
   # ifconfig tap0 create
3. turn on device:
   # ifconfig tap0 up

The tap0 device will be used only by new bhyve vm. If we want to allow a virtual machine to access the network or access our NFS server we need to bridge it with one of our physical interfaces (or vnet). Again:

1. load bridge module:
   # kldload if_bridge
2. create bridge:
   # ifconfig bridge0 create
3. add members to a bridge (where physical_interface is for example em0):
   # ifconfig bridge0 addm ${physical_interface} addm tap0
4. turn on a bridge:
   # ifconfig bridge0 up

Configuring userboot
Unfortunately userboot, which is used as a bootloader, is unable to set a boot variables. Those variables contains information which the network interface needs to be configured or where to mount the root filesystems from. We need to explicitly set those variables in bootloader. Our interface configuration could look something like that:
boot.netif.name=vtnet0
boot.netif.hwaddr=02:01:02:03:04:05
boot.netif.ip=192.168.67.2
boot.netif.netmask=255.255.255.0
Now we need to go to the second part which is to tell userboot from what filesystem it should mount the root from. For this we need a NFS handle which contains all the information used to uniquely identify exported files. NFS handle is built from several things like inode of file, the file-system identifier and generation identifier. Getting this handler isn’t easy. But this time we are lucky. Stefan Bethke wrote a small hack which allows us to get a NFS handler (github). So we can easy run now:
# ./boot_root_nfs [server:/exported/root] [mountpoint]# ./boot_root_nfs 192.168.67.1:/exported/root /
And output should look similar to:
boot.nfsroot.server=192.168.67.1
boot.nfsroot.nfshandle=X9211d0asddbf41180a00026009000000a1asd13000000000000000000X
boot.nfsroot.nfshandlelen=28
boot.nfsroot.path=/path
If we don’t change the directory in which we have root (inode) we can use the same NFS handler multiple times. All those option can be passed to the bhyveload -e options or we can put this configuration to the boot/loader.rc file in virtual machine with.

Running VM
The last step is to load FreeBSD kernel:
# bhyveload -e boot.netif.name=vtnet0 \
  -e boot.netif.hwaddr=02:01:02:03:04:05 \
  -e boot.netif.ip=192.168.67.2 \
  -e boot.netif.netmask=255.255.255.0 \
  -e boot.nfsroot.server=192.168.67.1 \
  -e boot.nfsroot.nfshandle=X9211d039debf41180a0002d011000000b5a047000000000000000000X \
  -e boot.nfsroot.nfshandlelen=28 \
  -e boot.nfsroot.path=/exported/root \
  -m[MEMORY_COUNT_ex_1024] \
  -h [WHERE_TO_LOOK_FOR_VM_BOOT_DIRECTORY] \
  VM_NAME
If we give all those options to the boot/loader.rc file then command should look more like this:
# bhyveload -m[MEMORY_COUNT_ex_1024] \
  -h [WHERE_TO_LOOK_FOR_VM_BOOT_DIRECTORY] \
  VM_NAME
Output should looks something like:
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(root@hostname, Thu Nov  5 02:29:25 CET 2015) /boot/kernel/kernel text=0x139f198 data=0x134448+0x4d8bb0 syms=[0x8+0x15bbb0+0x8+0x17765f]

And finally to run virtual machine:
# bhyve -s 0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -m[MEMORY_COUNT] -A -H -P -l com1,stdio VM_NAME
This command creates a virtual machine with a host bridge emulation, lpc emulation (com1), network interface (tap0). Other options you can find in the manual for bhyve command.