OpenBSD/NetBSD on FreeBSD using grub2-bhyve

Sept. 16, 2018, 6:24 p.m.

When I was writing a blog post about the process title, I needed a couple of virtual machines with OpenBSD, NetBSD, and Ubuntu. Before that day I mainly used FreeBSD and Windows with bhyve. I spent some time trying to set up an OpenBSD using bhyve and UEFI as described here. I had numerous problems trying to use it, and this was the day I discovered the grub2-bhyve tool, and I love it!

The grub2-bhyve allows you to load a kernel using GRUB bootloader. GRUB supports most of the operating systems with a standard configuration, so exactly the same method can be used to install NetBSD or Ubuntu. First, let's install grub2-bhyve on our FreeBSD box:
# pkg install grub2-bhyve

To run grub2-bhyve we need to provide at least the name of the VM. In bhyve, if the memsize is not specified the default VM is created with 256MB of the memory.
# grub-bhyve test
                                          GNU GRUB version 2.00
   Minimal BASH-like line editing is supported. For the first word, TAB lists possible command
   completions. Anywhere else TAB lists possible device or file completions.


After running grub-bhyve command we will enter the GRUB loader. If we type the ls command, we will see all the available devices. In the case of the grub2-bhyve there is one additional device called "(host)" that is always available and allows the host filesystem to be accessed. We can list files under that device.
grub> ls
grub> ls (host)/
libexec/ bin/ usr/ bhyve/ compat/ tank/ etc/ boot/ net/ entropy proc/ lib/ root/ sys/ mnt/ rescue/ tmp/ home/ sbin/ media/ jail/ COPYRIGHT var/ dev/

To exit console simply type ‘reboot'. I would like to install my new operating system under a ZVOL `ztank/bhyve/post`. On another terminal, we create:
# zfs create -V 10G ztank/bhyve/post


If you don't use ZFS for some crazy reason you can also create a raw blob using the truncate(1) command.
# truncate -s 10G post.img

I recommend installing an operating system from the disk image (installXX.fs for OpenBSD and NetBSD-X.X-amd64-install.img for NetBSD). Now we need to create a device map for a GRUB.
cat > /tmp/ << EOF
(hd0) /directory/to/disk/image
(hd1) /dev/zvol/ztank/bhyve/post

The mapping files describe the names for files in the GRUB. In our case under hd0 we will have an installation image and in hd1 we will have our ZVOL/blob. You can also try to use an ISO image then instead of using hd0 device name use a cd0. When we will run the grub-bhyve command we will see two additional devices.
# grub-bhyve -m /tmp/ post
grub> ls
(hd0) (hd0,msdos4) (hd0,msdos1) (hd0,openbsd9) (hd0,openbsd1) (hd1) (host)

The hd0 (in this example OpenBSD image) contains multiple partitions. We can check what is on it.
grub> ls (hd0,msdos4)/
boot bsd 6.4/ etc/

And this is the partition that contains a kernel. Now we can set a root device, load an OpenBSD kernel and boot:
grub> set root=(hd0,msdos4)
grub> kopenbsd -h com0 -r sd0a /bsd
grub> boot

After that, we can run bhyve virtual machine. In my case it is:
# bhyve -c 1 -w -u -H \
               -s 0,amd_hostbridge \
               -s 3,ahci-hd,/directory/to/disk/image \
               -s 4,ahci-hd,/dev/zvol/ztank/bhyve/post \
               -s 31,lpc -l com1,stdio \

Unfortunately explaining the whole bhyve(8)  command line is beyond this article. After installing the operating system remove hd0 from the mapping file and the image from the bhyve(8) command. If you don't want to type all those GRUB commands, you can simply redirect them to the standard input.
cat << EOF | grub-bhyve -m /tmp/ -M 512 post
set root=(hd0,4)
kopenbsd -h com0 -r sd0a /bsd