Update UEFI boot code after ZFS upgrade on FreeBSD

Whenever a new FreeBSD version supports new features for ZFS, an upgrade of the existing pools is necessary. This is indicated when the status of a pool is displayed:

# zpool status zroot
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: none requested

	zroot        ONLINE       0     0     0
	  da0p2.eli  ONLINE       0     0     0

errors: No known data errors

Performing the update is pretty straight forward, the necessary command is already listed. However, when the pool the system is booted from is updated, the following message appears:

# zpool upgrade -a
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':

If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
the following command will do it:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

The suggested command to update the boot code, however, only applies to systems still booting with the old BIOS. If you use it on a system booting with UEFI, the UEFI partition will be overwritten and is not recognized as bootable anymore.

The proper command to use on an UEFI system is the following:

# gpart bootcode -p /boot/boot1.efifat -i 1 da0

Make sure to apply this to all disks of the boot pool.

Limit SSH to IPv6 on FreeBSD

When I first set up my server, I got numerous daily attempts to login via SSH almost immediately. None of them were successful, because I use public key authentication, of course, but the log spam was annoying nonetheless. The problem is with only 232 addresses, scanning all possible IPv4 addresses for open ports is quite easy.

But, yet again, IPv6 to the rescue. It has 2128 possible addresses, so scanning all of them for open ports is doable but takes a significant amount of time. My server has a /64 subnet, so even if someone knows the prefix, there are still 264 addresses to scan. Both my home internet and my cell phone provider support IPv6 (as they should), so limiting SSH to a spare IPv6 address was the obvious solution.

This is not an approach for security, but merely to get rid of the annoying login attempts. The server still has to be secured. No one should believe that trying to “hide” the IP address does anything for security. If anyone really wants to find it, he will. But it helped to reduce the random failed login attempts, from a couple of hundreds on bad days, to not a single one in more than a year. So without all this noise in the logs, I can more easily see if anyone is actually trying to get into my server now.

The first step is to add another IPv6 address to the network interface, which can be used for SSH. Let’s assume the IPv6 prefix for the /64 subnet is 2001:10:20:30. All regular services might use 2001:10:20:30::1, so 2001:10:20:30::2222 could be reserved for SSH only. The additional address can be added in the /etc/rc.conf, assuming the network interface is em0:

ifconfig_em0_ipv6="inet6 2001:10:20:30::1/64"
ifconfig_em0_alias0_ipv6="inet6 2001:10:20:30::2222/64"

The networking can be restartet with:

# service netif restart

Alternatively, the server can also be rebooted. Now ifconfig should list the additional address:

# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 96:00:00:aa:bb:cc
hwaddr 96:00:00:aa:bb:cc
inet6 2001:10:20:30::1 prefixlen 64 
inet6 fe80::9400:ff:feaa:bbcc%vtnet0 prefixlen 64 scopeid 0x1 
inet6 2001:10:20:30::2222 prefixlen 64 
inet netmask 0xffffffff broadcast 
media: Ethernet 10Gbase-T <full-duplex>
status: active

You can also try to ping the new address and it should work fine. If the services on your server are not bound to specific IP addresses, but to any, now is the time to fix that. The other services should only use the 2001:10:20:30::1 address, and 2001:10:20:30::2222 is for SSH exclusively. sockstat is your friend.

With the new IPv6 address up and running, SSH can be configured to only listen to it and nothing else. This is done by having only a single ListenAddress parameter in /etc/ssh/sshd_config:

ListenAddress 2001:10:20:30::2222

Save the config file and be sure you can access your server directly or via a serial console, in case you made a mistake and lock yourself out. Then restart SSH to apply the changes:

# service ssh restart

You can check with sockstat if sshd is only listening to the new IPv6 address:

# sockstat -l | grep sshd
root sshd 34490 3 tcp6 2001:10:20:30::2222:22 *:*

If it is listening to anything else, check your sshd_config file.

The only way to access the server via SSH is by using the new dedicated IPv6 address:

$ ssh 2001:10:20:30::2222

This is inconvenient, so I recommend adding a new subdomain, like myssh.myserver.com, with a single AAAA record:

$ host myssh.myserver.com
myssh.myserver.com has IPv6 address 2001:10:20:30::2222

With these little changes the amount of random login attempts can be significantly reduced. However, it only works when all of your devices from which you manage the server support and use IPv6. But that should be the standard nowadays.

Backup FreeBSD to an encrypted image on a network share with rsync

While there are many fancy backup tools, often a simple daily rsync may already be sufficient. The advantage of an rsync backup is that it’s file system based, so no special tools are required to browse and access the files in the backup. However, rsync only copies the files, so there is no security measure like encryption. That’s fine locally or within a private network, but sometimes you only have a backup space you can’t trust, like a share on another machine or cloud storage.

Luckily, FreeBSD comes with everything necessary to create an encrypted image to use rsync with. This not only encrypts the backup, but also the copying process, because the encryption is performed before the data is written over the network. In the following, I’ll assume the backup share is mounted locally on /backup. It doesn’t really matter if via SMB, AFP, NFS, or anything else, as long as it can be mounted.

First we need to create an image that is large enough to hold the backup files:

# dd if=/dev/random of=/backup/backup.img bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 6.602198 secs (162633995 bytes/sec)

This will create a 1 GB image, because the block size of 1 MB will be copied 1024 times. Adjust count as necessary. I recommend using /dev/random instead of /dev/zero. While the latter is faster, it will also reveal how much space of the image is in use, since everything not overwritten will still be zero.

Now we can create a vnode for the image, which creates a block device:

# mdconfig -a -t vnode -f /backup/backup.img -u 0

This will create the block device /dev/md0 for the image file. The -u parameter can also be omitted to automatically use the next available index. This block device can be used like any other disk, so it can be encrypted with geli with a key and/or a passphrase. To make scripting easier, I’ll only use a key. Since it’s stored on the machine to create the backup from, a malicious user being able to read it already has access to every data the backup includes. We don’t need to memorize the key, so it can be random:

# dd if=/dev/random of=/usr/local/etc/backup.key bs=4096 count=1

A length of 256 bytes is more than enough, but for the extra paranoia I’ll use 4096 bytes. With the key, the geli layer for the image can be initialized:

# geli init -l 256 -K /backup/backup.key -P /dev/md0

Metadata backup can be found in /var/backups/md0.eli and
can be restored with the following command:

        # geli restore /var/backups/md0.eli /dev/md0

The default algorithm is AES-XTS, which is fine and has hardware support in modern CPUs. The default AES key length is 128 bits, but increasing it to 256 bits with -l 256 can’t hurt. The argument of parameter -K points to the key file and -P disables the passphrase. The following activates the encryption layer for the image device:

# geli attach -k /usr/local/etc/backup.key -p /dev/md0

This time, the argument of -k points to the key file and -p disables the passphrase. As a result, we now have another block device called /dev/md0.eli. It can be used just like the original /dev/md0, but all data passing through it will be encrypted and decrypted, before written to and read from /dev/md0, and therefore the image file. At this point we have encrypted access to the image file like a normal but still empty disk. Every disk needs a file system, so does our image:

# newfs /dev/md0.eli
/dev/md0.eli: 1024.0MB (2097144 sectors) block size 32768, fragment size 4096
        using 4 cylinder groups of 256.00MB, 8192 blks, 32768 inodes.
super-block backups (for fsck_ffs -b #) at:
 192, 524480, 1048768, 1573056

It is also possible to create a partition table with multiple partitions and file systems, but for the sake of simplicity I’ll just use the entire image with UFS2. Now it’s time to mount:

# mount /dev/md0.eli /mnt

That’s pretty much it! Now simply use rsync or and even cp to backup files. Here is an example for rsync:

# rsync -a --delete --exclude '/mnt' --exclude '/dev' \
  --exclude '/backup' / /mnt/

This creates a full recursive backup (-a) of / to /mnt and deletes files in the destination directory that have been deleted in the source directory since the last backup (–delete). To avoid loops and errors when trying to access the devices in /dev, some excludes should be added.

After the backup is finished, the image file can be unmounted, the geli layer detached and the vnode destroyed:

# umount /dev/md0.eli
# geli detach /dev/md0
# mdconfig -d -u 0

The steps necessary for a backup script, which can be scheduled via cron, are creating the vnode, attaching the geli layer, mounting, performing the backup, and cleaning up nicely:

mdconfig -a -t vnode -f /backup/backup.img -u 0
geli attach -k /usr/local/etc/backup.key -p /dev/md0
mount /dev/md0.eli /mnt
rsync -a --delete --exclude '/mnt' --exclude '/dev' --exclude '/backup' / /mnt/
umount /dev/md0.eli
geli detach /dev/md0
mdconfig -d -u 0

Scripting an automated backup for an AVM FritzBox router

My home server is set up to backup pretty much every device in my network automatically every day. So I figured why not also backup my AVM FritzBox router. In case it breaks or an update fails and it has to be reset, I can easily restore a backup with all settings, phone book entries, logs etc.

Fortunately, there is a very basic interface for automation, so it is possible to shell script the process of exporting the entire configuration. The result is identical to the manual export in the administration interface to export everything to a backup file. The necessary steps are as follows:

  1. Request a challenge
  2. Log in by responding to the challenge
  3. Extract session ID
  4. Export the backup file
  5. Log out

For sake of simplicity, I will assume that the router is available at https://fritz.box (which it is by default) and that the admin password is supersecret.

A challenge can be requested with the URL https://fritz.box/login_sid.lua:

curl -s -k https://fritz.box/login_sid.lua

The response is a SessionInfo XML, which looks like this:

<?xml version="1.0" encoding="utf-8"?>

The challenge is a random string that changes every time the URL is called. Without providing the challenge response, the session ID (SID) will remain empty. To extract the challenge and store it in a variable, simply add a sed statement:

CHALLENGE=`curl -s -k https://fritz.box/login_sid.lua | \
sed 's/.*<Challenge>\([a-z0-9]*\)<.*/\1/'`

So now we have to construct the response and call the URL again. The response is <challenge>-md5(<challenge>-<password>), but it is important to make sure the encoding is UTF-16LE before calculating the hash. Otherwise it will differ and won’t be accepted. This can be done by including iconv:

RESPONSE=$CHALLENGE-`echo -n "$CHALLENGE-$PASSWORD" | iconv -t UTF-16LE | md5`

Call the login URL again, this time with the response:

curl -s -k --data $DATA https://fritz.box/login_sid.lua

In case you want to use a different user than admin, just provide the username here. The response is again the SessionInfo XML, but SID should be set. It also lists the rights the now logged in user has:

<?xml version="1.0" encoding="utf-8"?>

In case the SID remains empty, the login failed. If the password is correct, but it still doesn’t work, it may be an encoding issue with the MD5 hash. Keep in mind that the encoding may also depend on the shell used. Use the following example to check your script:

$ echo -n "test" | iconv -t UTF-16LE | md5

To extract the SID and store it in a variable, add the following sed statement:

SID=`curl -s -k --data $DATA https://fritz.box/login_sid.lua | \
sed 's/.*<SID>\([a-z0-9]*\)<.*/\1/'`

This SID can now be used with the call to export all settings for backup. To do this, a form has to be sent to https://fritz.box/cgi-bin/firmwarecfg, including the SID, the desired operation, as well as the password required to restore the backup in this case. I’ll just use the password backup. This is necessary, because the export contains sensitive information, like ISP and SIP accounts, which will be encrypted.

curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
--form ConfigExport= https://fritz.box/cgi-bin/firmwarecfg

Now the only thing left to do is log out, so the session doesn’t stay open unnecessarily:

curl -s -k --data $DATA https://fritz.box/login_sid.lua

That’s it! The exported file includes all information stored in the FritzBox, so in case anything goes wrong, it allows a complete restore.

Using a Raspberry Pi as a serial console server for FreeBSD

My home server runs FreeBSD 11.0, and, of course, has full disk encryption. This requires a passphrase to be entered while booting, so I need a keyboard and monitor attached, and have to be in the same room for restarts and after power outages. This is not acceptable at all. There is no need for a display and keyboard otherwise, and I want to be able to enter the passphrase remotely, but without compromising security. Luckily, the server has a serial port and I have a spare Raspberry Pi. So I can connect the serial port to the Raspberry Pi, SSH into it and use the serial console of the server just like physical access. Hooray for stuff from the 1960s still being useful!

So first things first, I need to connect the Raspberry Pi to the serial console. Since it only has USB (aside from GPIO), a USB to serial port adapter is necessary. The Raspberry Pi runs FreeBSD 11.0, which has good out-of-the-box support for the Prolific PL2303 chipset, so I just ordered a cheap adapter based on it. It probably works with Linux as well, if you’re using Raspbian. You can use dmesg to check if it was detected correctly:

# dmesg | grep Prolific
ugen0.4: <Prolific Technology Inc.> at usbus0
uplcom0: <Prolific Technology Inc. USB-Serial Controller, class 0/0,
rev 2.00/3.00, addr 4> on usbus0

Don’t forget (like I did) that the USB to serial port adapter only provides you with a serial port. To connect it to the server’s serial port, you still need a null modem cable or adapter.

Now that we have the hardware, it’s time to set up the server to use the serial console. This is pretty straight-forward, just add the following to your /boot/loader.conf and reboot, as described in the FreeBSD handbook:


If you don’t use EFI yet, replace efi with vidconsole. This outputs all kernel messages also to the serial console and lets you type in the passphrase during the boot process. You can also only use comconsole, but then there won’t be much output on a real monitor in case you need one.

Additionally, you may like to see a login prompt after the boot process is done, in case the server’s network is down and SSH doesn’t work. A terminal on the serial port can be activated by adding the following line in /etc/ttys:

ttyu0   "/usr/libexec/getty std.9600" dialup  on secure

Now you should already be able to use the serial console via the Raspberry Pi, with a tool of your choice:

# tip ucom1
# screen /dev/cuaU0 9600
# cu -l /dev/cuaU0 -s 9600

If you don’t want to be root to access the console (you don’t), you can add your user to the group dialer:

# pw groupmod dialer -M solence

Now that all is up and running, be aware that the serial console works like a monitor. If you log in via serial console and only close serial connection, it’s just like turning off your monitor, you’re still logged in! So be aware to always log out before closing the serial connection.


Increasing the port speed

This setup is usable, but painfully slow, because the default port speed is still 9,600 baud. In case you don’t remember the ancient unit baud, it’s symbols per second. This equals bits/sec, so 9,600 baud convert to 1,200 bytes/sec. Considering that the standard terminal line has 80 characters, that’s only 15 lines/sec.

So increasing the serial port speed to the maximum of 115,200 baud would be a good idea. This equals 14,000 bytes/sec or 180 terminal lines/sec, which sounds much better. Some tweaking on the server’s side is required, however.

First, add the desired speed to /boot/loader.conf:


Then change the speed in the corresponding entry in /etc/ttys:

ttyu0   "/usr/libexec/getty std.115200" dialup  on secure

In theory, this is sufficient and only requires a reboot to work, but in my case I also had to configure the serial port device. Check with stty if the port has the correct speed on the server:

# stty -f /dev/ttyu0

If the speed is set to 115,200 baud, everything is fine. If it’s still 9,600 baud, it has to be adjusted like described in this great tutorial.

After that’s done, the serial console can be used with the increased speed on the Raspberry Pi:

$ screen /dev/cuaU0 115200
$ cu -l /dev/cuaU0 -s 115200

You can also add an alias in /etc/remote for easy use with tip on the Raspberry Pi:


Now you can just use:

$ tip server

If your Raspberry Pi is accessible remotely, directly or via VPN, power outages or remote maintenance are no problem anymore. But make sure you can still reach the Raspberry Pi when the server you connected it to is down, otherwise all this would be pointless. Also, now you don’t even need keyboard and monitor at all, because it’s way more convenient to use SSH and the serial console than getting up and moving over to the box 🙂

Installing and updating FreeBSD 11.0 release on a Raspberry Pi

FreeBSD runs great on a Raspberry Pi, but the official images are all stable or current branches and there is no support for binary updates, i.e. freebsd-update. So I tried to figure out how to install the latest release version and keep it updated as well, without starting from scratch with a new image every time.

While it is possible to build a kernel and world on the Raspberry Pi itself, it takes about a week to finish (trust me, I tried) and you need a much larger /tmp on a disk or share for it to work in the first place. Since FreeBSD can also be cross-compiled, I decided to use my Xeon box to do the heavy lifting and then just install it on the Raspberry Pi.

So let’s get started with getting the source and building it with armv6 as a target architecture on an amd64 system. I assume that /usr/src is empty, so first get the source. We want the Release Engineering (RelEng) branch, which is the latest release version, including security updates:

# cd /usr/src
# svn checkout https://svn.freebsd.org/base/releng/11.0

After getting the code, we can keep it updated with svn update. I recommend subscribing to the FreeBSD-Announce mailing list, to get a notification every time there is an update.

Now we can build the world and kernel, but with armv6 for the Raspberry Pi as the target architecture:

# make TARGET_ARCH=armv6 UBLDR_LOADADDR=0x2000000 buildworld
# make TARGET_ARCH=armv6 KERNCONF=RPI-B buildkernel

Please note that the UBLDR_LOADADDR variable has to be set, otherwise the kernel won’t boot. Also, the kernel configuration is for the first Raspberry Pi, for the newer Raspberry Pi 2 use KERNCONF=RPI2. This takes a while, but it can be sped up by using the -j<n> parameter with make, where n equals the number of CPU cores + 1, e.g. -j5 for a quad core. My Xeon with an SSD needs about 30 minutes to build the world and kernel.

There are basically several ways to install everything on the Raspberry Pi, but I could only figure out to get one working. The most convenient would be to mount /usr/src and /usr/obj via NFS on the Raspberry Pi and install everything locally. That didn’t work for me. The compiled files weren’t executable, probably because it’s a cross-compiled and not a native build. Another method is to mount / of the Raspberry Pi on the build box via NFS (yay security) and install from the build box. That did work for the kernel, but not for world. The problem is that most files are in use and therefore can’t be overwritten. That’s why the FreeBSD handbook advises to use single user mode, but then there is no networking. Also my USB keyboard wasn’t recognized early enough in the boot process to enter single user mode. So in the end, I just pulled the SD card and put it in a card reader plugged into the build box. This also turned out to be much faster than via network. Mounting the SD card locally is as simple as it can be:

# mount /dev/da0s2a /mnt

Then install kernel and install world can be done like in the handbook, but with a few added variables and parameters:

# make TARGET_ARCH=armv6 KERNCONF=RPI-B DESTDIR=/mnt installkernel
# mergemaster -p -A armv6 -D /mnt
# make TARGET_ARCH=armv6 DESTDIR=/mnt installworld
# mergemaster -iF -A armv6 -D /mnt
# make TARGET_ARCH=armv6 DESTDIR=/mnt delete-old
# make TARGET_ARCH=armv6 DESTDIR=/mnt delete-old-libs

That’s it! Now unmount the SD card, plug it into the Raspberry Pi again, and be greeted with the current release version with the lastest patch:

# uname -a
FreeBSD raspberrypi 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #3 r315180: Sun
Mar 12 23:15:58 CET 2017     solence@raspberrypi:/usr/obj/arm.armv6/usr/
src/sys/RPI-B  arm

# freebsd-version

IPv6 with Hurricane Electric 6in4 tunnel on DD-WRT

My Asus WL500g Deluxe router is running an Eko build of DD-WRT, which comes with IPv6 support. They haven’t yet included it in the official releases for some reason. However, there is still no ISP offering native IPv6 here. It’s a shame. So some sort of tunnel is needed to finally equip my LAN with an IPv6 subnet.

I didn’t really trust those tunnel brokers, so I tried a 6to4 tunnel. There is a good tutorial and only a few scripts are needed to get it running. The problem was that my IPv4 address is dynamic and changes with every reconnect, which is also enforced every 24h. Unfortunately with 6to4 this also changes the IPv6 prefix since it includes the v4 address. That’s no fun and it also messes up the routing and clients using automatic address assignment. Ok, I need a tunnel broker.

My first try was SIXXS, which took like forever to get everything approved. After finishing the ‘paperwork’, my very own /48 subnet was ready. Some hours later I figured out how to run their proprietary aiccu client on my router. It basically worked, but the connection was slow and the client crashed again and again. The port for MIPS-based routers was from 2007 and I couldn’t find a more recent version. It was very disappointing. So hopefully Hurricane Electric will do better.

The tunnel request is pretty easy and instant. You don’t have to wait a week or two until everything is approved. There is no special software, so you have to rely on the tunneling features of your operating system, that is good. DD-WRT has no GUI to configure tunnels, so some scripts are necessary. The IPv6 tutorial for HE tunnels on DD-WRT is not very good, but I finally figured out how it works. To my surprise this solution is very stable and also very fast. The latency with IPv6 is sometimes even lower than with native IPv4. They must have a really good routing.

So here’s what needs to be done:

  • Get an Eko build running for IPv6 support. For my Asus router it’s the VINT build in V24_TNG.
  • Go to Administration and Management and under IPv6 Support, enable IPv6, Radvd and use the following for Radvd config:
    interface br0 {
       MinRtrAdvInterval 3;
       MaxRtrAdvInterval 10;
       AdvSendAdvert on;
       prefix <Your subnet prefix>::/64 {
          AdvOnLink on;
          AdvAutonomous on;

    This enables Router advertisements, so all the clients can assign an address automatically. Note that this only works with /64 subnets. If you have an /48 subnet you can only use one /64 part of it. Make sure that br0 really is your LAN interface and replace it with the correct identifier otherwise.

  • Go to Commands under Administration and add the following Startup Script:
    /usr/bin/wget 'http://ipv4.tunnelbroker.net/ipv4_end.php?
    ipv4b=AUTO&pass=<your password as MD5 hash>&
    user_id=<your user id>&tunnel_id=<your tunnel id>' -O -
    if [ -n `ifconfig | grep 'he-ipv6'` ]
       insmod ipv6
       sleep 3
       ip tunnel add he-ipv6 mode sit remote <HE IPv4 tunnel endpoint> ttl 64
       ip link set he-ipv6 up
       ip addr add <your IPv6 tunnel endpoint>/64 dev he-ipv6
       ip route add ::/0 dev he-ipv6 metric 1
       ip -6 addr add <your router's IPv6 address>/64 dev br0
    radvd -C /tmp/radvd.conf &

    I have an USB stick attached, and placed everything above in a script file. Makes it a little clearer. Make sure the wget call has no line breaks like displayed here (due to limited space).

  • Add the following Firewall rules on the same page:
    iptables -I INPUT 2 -p ipv6 -i br0 -j ACCEPT
    iptables -I INPUT 2 -p ipv6 -i ppp0 -j ACCEPT

That’s it already. Reboot the router and IPv6 should work fine on the router itself and on clients which automatically assigned themselves addresses of the subnet announced by radvd.

For increased reliability, I also added a script which is run as a cron job every couple of minutes:

IP=<HE IPv6 tunnel endpoint>

RESULT=`ping6 -c 3 $IP | grep '64 bytes'`

if [ -n $RESULT ]
  /usr/bin/wget 'http://ipv4.tunnelbroker.net/ipv4_end.php?
  ipv4b=AUTO&pass=<your password as MD5 hash>&
  user_id=<your user id&tunnel_id=<your tunnel id>' -O -

This tries to ping the tunnel endpoint of HE three times. If it fails, the website to update the IP address is called, because probably the IPv4 address changed without the Startup Script being run.