I love FreeNAS. Its awesome, well built, well-supported. But as my needs increased, I wanted to use my FreeNAS box for more than the basics. In particular, I was moving towards a single host to run as a:

1. Family NAS server
2. Development server
3. IRC client
4. VM server
5. Web server
6. Email Server
7. Git Server
8. Home Firewall
9. Home IPv6 gateway
10. IPv6 VPN and Jump box

FreeNAS could easily do all of this. But I found myself using the device for everything but a NAS server. Also, as my experience on FreeBSD reaching proficient-status, I wanted to jump in the deep end and manually configure a production system from scratch. So I thanked FreeNAS for their contribution, yanked out the USB disks and installed FreeBSD 11.1 on a separate USB disk.

During installation, I was careful not to touch the /dev/ada devices, as that would destroy my precious files. Instead, I installed to the second USB disk, /dev/da1, while the installation medium was /dev/da0. This was obviously a problem, because at reboot the USB disk would become /dev/da0 and the kernel would panic upon not finding a /dev/da1. So I dropped to the terminal and mounted zroot/ROOT/default volume, which is the / directory, to /tmp/root as follows.

zfs set mountpoint=/tmp/root zroot/ROOT/default
zfs mount zroot/ROOT/default

Then I edited /tmp/root/etc/fstab and changed /dev/da1p2 to /dev/da0p2, umounted, reset the machine and FreeBSD booted without a glitch.

As mentioned, I plan on using this system fairly heavily going forward so the 8 GB USB disk would definitely not be sufficient. FreeBSD has an amazing feature where it isolates the base system from any user-installed applications or configurations. Rather than using symlink magic, my strategy was to store all application data on my two 4TB NAS disks.

First things first, I imported the pool as follows:

zpool import -f tank

The -f flag was necessary because for whatever reason ZFS thought tank was currently utilized. A quick zfs list revealed that FreeNAS had been mounting my disks to /tank. Unfortunately, the /tank directory is not utilized by default by FreeBSD. Therefore, I renamed each ZFS volume to a new /usr/local as follows. First, I created a zfs volume for tank/usr/share as follows.

zfs create tank/usr/local

Then I renamed the old paths to map to my new intended directory structure, as follows

zfs rename tank/old/path tank/usr/local/new/path
zfs set mountpoint=/usr/local/new/path tank/usr/local/new/path

This took a bit of time, but after completing these for all partitions, I ran:

zfs mount -a

With that, all ZFS shares were mounted as /usr/local subdirectories. All of my data was successfully migrated over without a single bit of data loss!

From here, I needed to re-create the jails. FreeNAS’s excellent jail web-based GUI allows you to create jails with their own independent network stack. This feature is called VIMAGE and is useful to isolate network services from the host FreeBSD system. VIMAGE is pre-compiled into the FreeNAS kernel. It is on by default on FreeBSD 12.0, but not 11.x and must be compiled in. To do this, you need to download and uncompress the src distribution, edit /usr/src/sys/amd64/conf/GENERIC and add in the following line:

options VIMAGE

Next, compile the kernel and install it as follows.

make -j 5 buildkernel
make installkernel

The -j 5 is because this machine is an i3 with 4 cores – feel free to adjust this depending on the number of cores you have.

With a successful reboot, I was now ready to migrate the jails over. I did so by moving the zfs jails volume to /usr/local/jail, such that my IRC client jail was /usr/local/jail/irc. Now the complicated part: Configuring the jails!

Since a jail using VIMAGE has a completely separate network stack, by default it renders a jail unable to communicate outside of itself. The way to allow communication you have to create an epair(4) pair and pass one side to the jail, as follows:

ifconfig epair create
ifconfig epair0a vnet JAILNAME

In this configuration epair0a would belong to the jail while epair0b would belong to the base FreeBSD host, such that they could communicate. But how to setup connectivity? I had a lot of options to have the jails connect outside, including:

• Being on the same subnet (192.168.1.0/24)
• Being on a separate VLAN from the rest of the network (might be the long-term plan)
• Have a single VLAN, have legacy IPv4 addresses identifiably different for ease, but have a single IPv6 network. I opted for this for now. Its simple and works.

This means creating an if_bridge(4) and attaching the network interface card, in my case an em(4) card and epairXb. Any frame to the bridge is relayed to the relevant epair(4). (Note, this not a route). I set my jail IP range as 192.168.100.0/24, just for organizational purposes. I also set the ISPs IP subnet to be 192.168.0.0/16, otherwise it would drop packets from 192.168.100.0/24. I am using TunnelBroker for my IPv6 traffic, as Verizon Fios does not offer IPv6. (As an side, this may be a good thing, since ISPs typically blocks ports, whereas TunnelBroker is completely unfiltered.) With that, Boom, network connectivity!

But…I wanted something repeatable per reboot, in the event of a power failure or loss. This meant I needed to go a little further. And here’s the complicated part. It took me about 4 hours to properly configure /etc/jail.conf:

/* Template */
host.hostname = "${name}.my.domain.prefix";

$ip4_route      = "192.168.100.1";
$ip6_route      = "IPV6PREFIX::1";

vnet;
vnet.interface = "epair${if}b";

persist;
allow.mount;
mount.devfs;
allow.sysvipc;

exec.prestart =  "ifconfig epair${if} create up";
exec.prestart += "ifconfig epair${if}a up";
exec.prestart += "ifconfig bridge0 addm epair${if}a up";

#exec.start += "/sbin/ifconfig epair${if}b up";
exec.start += "/sbin/ifconfig epair${if}b inet  ${ip4_addr}/24 up";
exec.start += "/sbin/ifconfig epair${if}b inet6 ${ip6_addr} prefixlen 64 up";

exec.start += "/sbin/route -4 add default ${ip4_route}";
exec.start += "/sbin/route -6 add default ${ip6_route}";

exec.start += "/sbin/ifconfig epair${if}b down";
exec.start += "/sbin/ifconfig epair${if}b up";

exec.start += "/bin/sh /etc/rc";

exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig bridge0 deletem epair${if}a";
exec.poststop = "ifconfig epair${if}a destroy";

irc {
        path = /usr/local/jail/irc;
	$if = "0";
	$ip4_addr 	= "192.168.100.2";
	$ip6_addr 	= "IPV6PREFIX::2";
}

www {
        path = /usr/local/jail/www;
	$if = "1";
	$ip4_addr 	= "192.168.100.3";
	$ip6_addr 	= "IPV6PREFIX::3";
}

In short, upon initialization, this creates a new epair(4) as specified by $if, attaches it to the jail, assigns the relevant IPv4/IPv6 information, and starts the init scripts. Shutdown is a mere detachment from the bridge and destruction of the epair(4). I also needed to assign the legacy IPv4 address to my em(4) interface.

Finally, I added the following sysctl(8) settings to /etc/sysctl.conf:

net.inet.ip.forwarding: 1
net.inet6.ip6.forwarding: 1

I did a lot of testing, reboot, restarting the jail, etc, and every time it worked. From the jails’ perspective, they didn’t even “know” they were migrated from one system to another. I wish I had tested if a FreeNAS plugin survived the migration, but I never used FreeNAS plugins anyways (what is this Plex I keep hearing about?).

Going forward, I plan:

• Place the jails on a properly separate VLAN to segment the network
• Consider use pfSense running in bhyve(8) to function as the Jail’s firewall of choice
• Look into vale(4) to replace if_bridge(4). But I can’t find any documentation on it!
• Figure out why TunnelBroker is failing on FreeBSD, but works just fine on my Linux Raspberry Pi – likely the fault of the ISP router.

My only regret: not installing HardenedBSD with LibreSSL.