Update UEFI boot code after ZFS upgrade on FreeBSD

Whenever a new FreeBSD version supports new features for ZFS, an upgrade of the existing pools is necessary. This is indicated when the status of a pool is displayed:

# zpool status zroot
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: none requested

	zroot        ONLINE       0     0     0
	  da0p2.eli  ONLINE       0     0     0

errors: No known data errors

Performing the update is pretty straight forward, the necessary command is already listed. However, when the pool the system is booted from is updated, the following message appears:

# zpool upgrade -a
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':

If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
the following command will do it:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

The suggested command to update the boot code, however, only applies to systems still booting with the old BIOS. If you use it on a system booting with UEFI, the UEFI partition will be overwritten and is not recognized as bootable anymore.

The proper command to use on an UEFI system is the following:

# gpart bootcode -p /boot/boot1.efifat -i 1 da0

Make sure to apply this to all disks of the boot pool.

Limit SSH to IPv6 on FreeBSD

When I first set up my server, I got numerous daily attempts to login via SSH almost immediately. None of them were successful, because I use public key authentication, of course, but the log spam was annoying nonetheless. The problem is with only 232 addresses, scanning all possible IPv4 addresses for open ports is quite easy.

But, yet again, IPv6 to the rescue. It has 2128 possible addresses, so scanning all of them for open ports is doable but takes a significant amount of time. My server has a /64 subnet, so even if someone knows the prefix, there are still 264 addresses to scan. Both my home internet and my cell phone provider support IPv6 (as they should), so limiting SSH to a spare IPv6 address was the obvious solution.

This is not an approach for security, but merely to get rid of the annoying login attempts. The server still has to be secured. No one should believe that trying to “hide” the IP address does anything for security. If anyone really wants to find it, he will. But it helped to reduce the random failed login attempts, from a couple of hundreds on bad days, to not a single one in more than a year. So without all this noise in the logs, I can more easily see if anyone is actually trying to get into my server now.

The first step is to add another IPv6 address to the network interface, which can be used for SSH. Let’s assume the IPv6 prefix for the /64 subnet is 2001:10:20:30. All regular services might use 2001:10:20:30::1, so 2001:10:20:30::2222 could be reserved for SSH only. The additional address can be added in the /etc/rc.conf, assuming the network interface is em0:

ifconfig_em0_ipv6="inet6 2001:10:20:30::1/64"
ifconfig_em0_alias0_ipv6="inet6 2001:10:20:30::2222/64"

The networking can be restartet with:

# service netif restart

Alternatively, the server can also be rebooted. Now ifconfig should list the additional address:

# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 96:00:00:aa:bb:cc
hwaddr 96:00:00:aa:bb:cc
inet6 2001:10:20:30::1 prefixlen 64 
inet6 fe80::9400:ff:feaa:bbcc%vtnet0 prefixlen 64 scopeid 0x1 
inet6 2001:10:20:30::2222 prefixlen 64 
inet netmask 0xffffffff broadcast 
media: Ethernet 10Gbase-T <full-duplex>
status: active

You can also try to ping the new address and it should work fine. If the services on your server are not bound to specific IP addresses, but to any, now is the time to fix that. The other services should only use the 2001:10:20:30::1 address, and 2001:10:20:30::2222 is for SSH exclusively. sockstat is your friend.

With the new IPv6 address up and running, SSH can be configured to only listen to it and nothing else. This is done by having only a single ListenAddress parameter in /etc/ssh/sshd_config:

ListenAddress 2001:10:20:30::2222

Save the config file and be sure you can access your server directly or via a serial console, in case you made a mistake and lock yourself out. Then restart SSH to apply the changes:

# service ssh restart

You can check with sockstat if sshd is only listening to the new IPv6 address:

# sockstat -l | grep sshd
root sshd 34490 3 tcp6 2001:10:20:30::2222:22 *:*

If it is listening to anything else, check your sshd_config file.

The only way to access the server via SSH is by using the new dedicated IPv6 address:

$ ssh 2001:10:20:30::2222

This is inconvenient, so I recommend adding a new subdomain, like myssh.myserver.com, with a single AAAA record:

$ host myssh.myserver.com
myssh.myserver.com has IPv6 address 2001:10:20:30::2222

With these little changes the amount of random login attempts can be significantly reduced. However, it only works when all of your devices from which you manage the server support and use IPv6. But that should be the standard nowadays.

Backup FreeBSD to an encrypted image on a network share with rsync

While there are many fancy backup tools, often a simple daily rsync may already be sufficient. The advantage of an rsync backup is that it’s file system based, so no special tools are required to browse and access the files in the backup. However, rsync only copies the files, so there is no security measure like encryption. That’s fine locally or within a private network, but sometimes you only have a backup space you can’t trust, like a share on another machine or cloud storage.

Luckily, FreeBSD comes with everything necessary to create an encrypted image to use rsync with. This not only encrypts the backup, but also the copying process, because the encryption is performed before the data is written over the network. In the following, I’ll assume the backup share is mounted locally on /backup. It doesn’t really matter if via SMB, AFP, NFS, or anything else, as long as it can be mounted.

First we need to create an image that is large enough to hold the backup files:

# dd if=/dev/random of=/backup/backup.img bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 6.602198 secs (162633995 bytes/sec)

This will create a 1 GB image, because the block size of 1 MB will be copied 1024 times. Adjust count as necessary. I recommend using /dev/random instead of /dev/zero. While the latter is faster, it will also reveal how much space of the image is in use, since everything not overwritten will still be zero.

Now we can create a vnode for the image, which creates a block device:

# mdconfig -a -t vnode -f /backup/backup.img -u 0

This will create the block device /dev/md0 for the image file. The -u parameter can also be omitted to automatically use the next available index. This block device can be used like any other disk, so it can be encrypted with geli with a key and/or a passphrase. To make scripting easier, I’ll only use a key. Since it’s stored on the machine to create the backup from, a malicious user being able to read it already has access to every data the backup includes. We don’t need to memorize the key, so it can be random:

# dd if=/dev/random of=/usr/local/etc/backup.key bs=4096 count=1

A length of 256 bytes is more than enough, but for the extra paranoia I’ll use 4096 bytes. With the key, the geli layer for the image can be initialized:

# geli init -l 256 -K /backup/backup.key -P /dev/md0

Metadata backup can be found in /var/backups/md0.eli and
can be restored with the following command:

        # geli restore /var/backups/md0.eli /dev/md0

The default algorithm is AES-XTS, which is fine and has hardware support in modern CPUs. The default AES key length is 128 bits, but increasing it to 256 bits with -l 256 can’t hurt. The argument of parameter -K points to the key file and -P disables the passphrase. The following activates the encryption layer for the image device:

# geli attach -k /usr/local/etc/backup.key -p /dev/md0

This time, the argument of -k points to the key file and -p disables the passphrase. As a result, we now have another block device called /dev/md0.eli. It can be used just like the original /dev/md0, but all data passing through it will be encrypted and decrypted, before written to and read from /dev/md0, and therefore the image file. At this point we have encrypted access to the image file like a normal but still empty disk. Every disk needs a file system, so does our image:

# newfs /dev/md0.eli
/dev/md0.eli: 1024.0MB (2097144 sectors) block size 32768, fragment size 4096
        using 4 cylinder groups of 256.00MB, 8192 blks, 32768 inodes.
super-block backups (for fsck_ffs -b #) at:
 192, 524480, 1048768, 1573056

It is also possible to create a partition table with multiple partitions and file systems, but for the sake of simplicity I’ll just use the entire image with UFS2. Now it’s time to mount:

# mount /dev/md0.eli /mnt

That’s pretty much it! Now simply use rsync or and even cp to backup files. Here is an example for rsync:

# rsync -a --delete --exclude '/mnt' --exclude '/dev' \
  --exclude '/backup' / /mnt/

This creates a full recursive backup (-a) of / to /mnt and deletes files in the destination directory that have been deleted in the source directory since the last backup (–delete). To avoid loops and errors when trying to access the devices in /dev, some excludes should be added.

After the backup is finished, the image file can be unmounted, the geli layer detached and the vnode destroyed:

# umount /dev/md0.eli
# geli detach /dev/md0
# mdconfig -d -u 0

The steps necessary for a backup script, which can be scheduled via cron, are creating the vnode, attaching the geli layer, mounting, performing the backup, and cleaning up nicely:

mdconfig -a -t vnode -f /backup/backup.img -u 0
geli attach -k /usr/local/etc/backup.key -p /dev/md0
mount /dev/md0.eli /mnt
rsync -a --delete --exclude '/mnt' --exclude '/dev' --exclude '/backup' / /mnt/
umount /dev/md0.eli
geli detach /dev/md0
mdconfig -d -u 0

Using a Raspberry Pi as a serial console server for FreeBSD

My home server runs FreeBSD 11.0, and, of course, has full disk encryption. This requires a passphrase to be entered while booting, so I need a keyboard and monitor attached, and have to be in the same room for restarts and after power outages. This is not acceptable at all. There is no need for a display and keyboard otherwise, and I want to be able to enter the passphrase remotely, but without compromising security. Luckily, the server has a serial port and I have a spare Raspberry Pi. So I can connect the serial port to the Raspberry Pi, SSH into it and use the serial console of the server just like physical access. Hooray for stuff from the 1960s still being useful!

So first things first, I need to connect the Raspberry Pi to the serial console. Since it only has USB (aside from GPIO), a USB to serial port adapter is necessary. The Raspberry Pi runs FreeBSD 11.0, which has good out-of-the-box support for the Prolific PL2303 chipset, so I just ordered a cheap adapter based on it. It probably works with Linux as well, if you’re using Raspbian. You can use dmesg to check if it was detected correctly:

# dmesg | grep Prolific
ugen0.4: <Prolific Technology Inc.> at usbus0
uplcom0: <Prolific Technology Inc. USB-Serial Controller, class 0/0,
rev 2.00/3.00, addr 4> on usbus0

Don’t forget (like I did) that the USB to serial port adapter only provides you with a serial port. To connect it to the server’s serial port, you still need a null modem cable or adapter.

Now that we have the hardware, it’s time to set up the server to use the serial console. This is pretty straight-forward, just add the following to your /boot/loader.conf and reboot, as described in the FreeBSD handbook:


If you don’t use EFI yet, replace efi with vidconsole. This outputs all kernel messages also to the serial console and lets you type in the passphrase during the boot process. You can also only use comconsole, but then there won’t be much output on a real monitor in case you need one.

Additionally, you may like to see a login prompt after the boot process is done, in case the server’s network is down and SSH doesn’t work. A terminal on the serial port can be activated by adding the following line in /etc/ttys:

ttyu0   "/usr/libexec/getty std.9600" dialup  on secure

Now you should already be able to use the serial console via the Raspberry Pi, with a tool of your choice:

# tip ucom1
# screen /dev/cuaU0 9600
# cu -l /dev/cuaU0 -s 9600

If you don’t want to be root to access the console (you don’t), you can add your user to the group dialer:

# pw groupmod dialer -M solence

Now that all is up and running, be aware that the serial console works like a monitor. If you log in via serial console and only close serial connection, it’s just like turning off your monitor, you’re still logged in! So be aware to always log out before closing the serial connection.


Increasing the port speed

This setup is usable, but painfully slow, because the default port speed is still 9,600 baud. In case you don’t remember the ancient unit baud, it’s symbols per second. This equals bits/sec, so 9,600 baud convert to 1,200 bytes/sec. Considering that the standard terminal line has 80 characters, that’s only 15 lines/sec.

So increasing the serial port speed to the maximum of 115,200 baud would be a good idea. This equals 14,000 bytes/sec or 180 terminal lines/sec, which sounds much better. Some tweaking on the server’s side is required, however.

First, add the desired speed to /boot/loader.conf:


Then change the speed in the corresponding entry in /etc/ttys:

ttyu0   "/usr/libexec/getty std.115200" dialup  on secure

In theory, this is sufficient and only requires a reboot to work, but in my case I also had to configure the serial port device. Check with stty if the port has the correct speed on the server:

# stty -f /dev/ttyu0

If the speed is set to 115,200 baud, everything is fine. If it’s still 9,600 baud, it has to be adjusted like described in this great tutorial.

After that’s done, the serial console can be used with the increased speed on the Raspberry Pi:

$ screen /dev/cuaU0 115200
$ cu -l /dev/cuaU0 -s 115200

You can also add an alias in /etc/remote for easy use with tip on the Raspberry Pi:


Now you can just use:

$ tip server

If your Raspberry Pi is accessible remotely, directly or via VPN, power outages or remote maintenance are no problem anymore. But make sure you can still reach the Raspberry Pi when the server you connected it to is down, otherwise all this would be pointless. Also, now you don’t even need keyboard and monitor at all, because it’s way more convenient to use SSH and the serial console than getting up and moving over to the box 🙂

Installing and updating FreeBSD 11.0 release on a Raspberry Pi

FreeBSD runs great on a Raspberry Pi, but the official images are all stable or current branches and there is no support for binary updates, i.e. freebsd-update. So I tried to figure out how to install the latest release version and keep it updated as well, without starting from scratch with a new image every time.

While it is possible to build a kernel and world on the Raspberry Pi itself, it takes about a week to finish (trust me, I tried) and you need a much larger /tmp on a disk or share for it to work in the first place. Since FreeBSD can also be cross-compiled, I decided to use my Xeon box to do the heavy lifting and then just install it on the Raspberry Pi.

So let’s get started with getting the source and building it with armv6 as a target architecture on an amd64 system. I assume that /usr/src is empty, so first get the source. We want the Release Engineering (RelEng) branch, which is the latest release version, including security updates:

# cd /usr/src
# svn checkout https://svn.freebsd.org/base/releng/11.0

After getting the code, we can keep it updated with svn update. I recommend subscribing to the FreeBSD-Announce mailing list, to get a notification every time there is an update.

Now we can build the world and kernel, but with armv6 for the Raspberry Pi as the target architecture:

# make TARGET_ARCH=armv6 UBLDR_LOADADDR=0x2000000 buildworld
# make TARGET_ARCH=armv6 KERNCONF=RPI-B buildkernel

Please note that the UBLDR_LOADADDR variable has to be set, otherwise the kernel won’t boot. Also, the kernel configuration is for the first Raspberry Pi, for the newer Raspberry Pi 2 use KERNCONF=RPI2. This takes a while, but it can be sped up by using the -j<n> parameter with make, where n equals the number of CPU cores + 1, e.g. -j5 for a quad core. My Xeon with an SSD needs about 30 minutes to build the world and kernel.

There are basically several ways to install everything on the Raspberry Pi, but I could only figure out to get one working. The most convenient would be to mount /usr/src and /usr/obj via NFS on the Raspberry Pi and install everything locally. That didn’t work for me. The compiled files weren’t executable, probably because it’s a cross-compiled and not a native build. Another method is to mount / of the Raspberry Pi on the build box via NFS (yay security) and install from the build box. That did work for the kernel, but not for world. The problem is that most files are in use and therefore can’t be overwritten. That’s why the FreeBSD handbook advises to use single user mode, but then there is no networking. Also my USB keyboard wasn’t recognized early enough in the boot process to enter single user mode. So in the end, I just pulled the SD card and put it in a card reader plugged into the build box. This also turned out to be much faster than via network. Mounting the SD card locally is as simple as it can be:

# mount /dev/da0s2a /mnt

Then install kernel and install world can be done like in the handbook, but with a few added variables and parameters:

# make TARGET_ARCH=armv6 KERNCONF=RPI-B DESTDIR=/mnt installkernel
# mergemaster -p -A armv6 -D /mnt
# make TARGET_ARCH=armv6 DESTDIR=/mnt installworld
# mergemaster -iF -A armv6 -D /mnt
# make TARGET_ARCH=armv6 DESTDIR=/mnt delete-old
# make TARGET_ARCH=armv6 DESTDIR=/mnt delete-old-libs

That’s it! Now unmount the SD card, plug it into the Raspberry Pi again, and be greeted with the current release version with the lastest patch:

# uname -a
FreeBSD raspberrypi 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #3 r315180: Sun
Mar 12 23:15:58 CET 2017     solence@raspberrypi:/usr/obj/arm.armv6/usr/
src/sys/RPI-B  arm

# freebsd-version