Migrating an existing FreeBSD installation to root on ZFS

There are plenty of guides and howtos on how to install FreeBSD onto a ZFS root.
It’s on the wiki at https://wiki.freebsd.org/RootOnZFS for example.
But I could not find any guides on how to migrate an existing installation to a ZFS root.
It turns out it wasn’t that difficult to achieve.

In my case I had a raidz(but this should work just as well for a mirror or raidz2 or 3) with three 2TB disks, but the root was on a UFS filesystem on a single 2,5″ HDD. So if the 2,5″ HDD would fail my server would not start. Not ideal.
My partition layout looked like this:

[root@nexus ~]# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ada3s1a 989M 444M 465M 49% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ada3s1e 989M 392k 909M 0% /tmp
/dev/ada3s1f 94G 3.4G 83G 4% /usr
/dev/ada3s1d 7.7G 2.0G 5.1G 29% /var
twotb/misc 2T 713G 1.3T 36% /misc
twotb/backup 1.4T 90G 1.3T 7% /misc/backup
twotb/movies 2.5T 1.3T 1.3T 50% /movies
twotb/timemachine 1.3T 51G 1.3T 4% /timemachine
twotb/tv 1.5T 198G 1.3T 13% /tv
twotb 1.3T 39k 1.3T 0% /twotb
twotb/home 1.3T 273M 1.3T 0% /usr/home
twotb/ports 1.3T 728M 1.3T 0% /usr/ports
twotb/distfiles 1.3T 1.3G 1.3T 0% /usr/ports/distfiles
twotb/mysql 1.3T 51M 1.3T 0% /var/db/mysql

And the disks in the zpool:

[root@nexus ~]# zpool status
NAME STATE READ WRITE CKSUM
twotb ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada2 ONLINE 0 0 0

I figured I should be able to move the root filesystem onto the raidz, and make FreeBSD boot off of the ZFS root.
Since I had added the raw disks to the raidz that meant I had no place to put a boot partition on. So first I needed to repartition each disk, and to do that I had to take them offline, one by one, fix the partitioning, and let them resilver.
To fix one disk, this is what I did(repeat for each disk, after the previous one has finished resilvering):

[root@nexus ~]# zpool offline twotb ada0
[root@nexus ~]# gpart destroy -F ada0
ada0 destroyed
[root@nexus ~]# gpart create -s gpt ada0
ada0 created
[root@nexus ~]# gpart add -b 40 -s 512k -t freebsd-boot ada0
ada0p1 added
[root@nexus ~]# gpart add -t freebsd-zfs -l disk0 ada0
ada0p2 added
[root@nexus ~]# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
bootcode written to ada0
[root@nexus ~]# zpool replace twotb ada0 /dev/gpt/disk0

After this was done to all the disks my setup still looked quite similar, the only visible change were the physical disk entries in the pool:

[root@nexus ~]# zpool status
NAME STATE READ WRITE CKSUM
twotb ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0

But this change is obviously important as the disks are now bootable, each and every one of them. They just dont have anything to boot, yet.
The next step is to copy all the data from the rootdisk to the raidz. But first we’ll create some new ZFS filesystems for it.

[root@nexus ~]# zfs create twotb/root && zfs create twotb/usr && zfs create twotb/var && zfs create twotb/tmp

Voila, we have now created partitions for each of the existing ones on the 2,5″ HDD. So lets copy the data to them.
I like to use rsync for this. You’ll notice I have some excludes since I already have some of the targets on the raidz.

[root@nexus ~]# rsync -av –exclude=ports –exclude=home /usr/ /twotb/usr/
[root@nexus ~]# rsync -av –exclude=db/mysql /var /twotb/var/
[root@nexus ~]# rsync -av –exclude=dev –exclude=var –exclude=usr –exclude=tv –exclude=movies –exclude=misc –exclude=timemachine –exclude=twotb –exclude=tmp / /twotb/root/

That should do the trick. Now we need to create the dev-folder in the new root. And then configure some mountpoints and boot properties.

[root@nexus ~]# mkdir /twotb/root/dev && chmod 555 /twotb/root/dev

Edit /twotb/root/boot/loader.conf and add:
zfs_load=YES
vfs.root.mountfrom=”zfs:twotb/root”

We don’t need the fstab anylonger since we handle the rootmounting in loader.conf, so lets clear it:
[root@nexus ~]# echo “” > /twotb/root/etc/fstab

Make sure you have the zpool.cache file under /twotb/root/boot/zfs/ (it should have been copied by the previous run of rsync).

Since we won’t have a swap partition, we need to create a swapfile. I’ll make mine 4GB.
[root@nexus ~]# dd if=/dev/zero of=/twotb/root/swapfile bs=1m count=4096

Add swapfile=”/swapfile” to /twotb/root/etc/rc.conf, and verify that you have zfs_enable=”YES” in there too.

Now lets fix the filesystem layout and bootfs properties

[root@nexus ~]# zfs umount twotb/var

[root@nexus ~]# zfs umount twotb/tmp

[root@nexus ~]# zfs umount twotb/root

[root@nexus ~]# zfs set mountpoint=/var twotb/var

[root@nexus ~]# zfs set mountpoint=/tmp twotb/tmp

[root@nexus ~]# zfs set mountpoint=/ twotb/root

[root@nexus ~]# zpool set bootfs=twotb/root twotb

You should now be able to reboot and either remove the existing rootdrive from the bios boot properties, or set the zfs drives to a higher priority.
Voila, you should now have a fully redundant 100% ZFS system.
Mine looks like this:
[root@nexus ~]# df -h
Filesystem Size Used Avail Capacity Mounted on
twotb/root 1.4T 5.2G 1.4T 0% /
devfs 1.0k 1.0k 0B 100% /dev
twotb/misc 2.1T 711G 1.4T 33% /misc
twotb/backup 1.5T 91G 1.4T 6% /misc/backup
twotb/movies 2.5T 1.1T 1.4T 44% /movies
twotb/timemachine 1.5T 52G 1.4T 4% /timemachine
twotb/tmp 1.4T 510k 1.4T 0% /tmp
twotb/tv 1.6T 196G 1.4T 12% /tv
twotb/usr 1.4T 3.1G 1.4T 0% /usr
twotb/home 1.4T 272M 1.4T 0% /usr/home
twotb/ports 1.4T 731M 1.4T 0% /usr/ports
twotb/distfiles 1.4T 1.3G 1.4T 0% /usr/ports/distfiles
twotb/var 1.4T 2.1G 1.4T 0% /var
twotb/mysql 1.4T 51M 1.4T 0% /var/db/mysql

This also makes it easier in the future with regards to updates/upgrades. You can take a snapshot or clone of the rootdrive, and if it doesn’t work as expected just revert to the old installation.

EDIT: In the first version of this document the size of the boot partition as 64K, that is however nowadays too small. FreeBSD 11 and newer require larger boot part. Therefore I updated this with 512k(do not make it any larger as it is not supported).

This entry was posted in FreeBSD, ZFS and tagged , , , , , , , , , . Bookmark the permalink.