Archives For freebsd

Dear Future Me,

I guess you came here googling for “FreeBSD PXE UEFI” trying to find out how to netboot your x86 dev box. Or arm64 box. Who knows what you’re hacking on in the future. To do that you need follow these simple steps:

  • Put loader.efi to tftpboot dir
  • Configure dhcpd along these lines:
    host amd64 {
            hardware ethernet  b8:ae:ed:77:88:99;
            filename "loader.efi";
            option root-path "/src/FreeBSD/tftproot/amd64";
            option routers;
  • Make sure root-path is in /etc/exports.
  • If you use MINIMAL-derived config add your NIC driver to /boot/loader.conf:

That’s pretty much it.

Take care

I spent Labor Day weekend laboring on VBox shared folders support for FreeBSD. It’s been some time since I worked on it last time so I had to refresh my memory first. Things have moved on since then – VBox in ports was updated to version 5, but fortunately Li-Wen synced up freebsd-vboxfs repo to the latest version. After three days of laid-back hacking I am glad to announce that following VOPs are kind of implemented (in no particular order): lookup, access, readdir, read, getattr, readlink, remove, rmdir, symlink, close, create, open, write. “Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host’s md5sum, create/remove directories, unzip zip file, etc but I doubt it would survive stress-test. Locking is all wrong at the moment and read/write VOPs allocate buffers for every operation.

I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them in vboxfs nodes, but if one of parent directories is renamed, all cached names should be modified accordingly. I am going to tackle these two problems once I have long enough stretch of time time sit and concentrate on task.

Looks like my attempt to cheap out on SSD for TK1 has backfired. I went for the cheapest SSD available in local store (Toshiba Q300) but when I tried to checkout FreeBSD sources to the drive I got bunch of WRITE_FPDMA_QUEUED timeouts and system locked up. The same thing happened when I tried to perform checkout on Linux. The drive itself was OK, it survived “svn co …/head” and dd when connected using USB-to-SATA adapter.

I believe the problem was that TK1 SATA voltage was out of Toshiba’s tolerance range. I replaced Q300 with Samsung EVO 850 and was able to checkout sources and finish buildworld using SSD for src/obj storage.

FreeBSD on Jetson TK1

June 28, 2016 — 4 Comments

I finally got around to BSDify my Jetson TK1. Here is short summary of what is involved. And to save you some scrolling here are artifacts obtained from whole ordeal:

Jetson TK1


First of all – my TK1 didn’t have U-Boot. Type of bootloader depends on the version of Linux4Tegra TK1 comes with. Mine had L4T R19, with some kind of “not u-boot” bootloader. My first attempt was to use tegrarcm tool, it uses libusb, so it’s possible to build it on FreeBSD with some elbow grease, but once I tried to run it – it gave me cryptic errors and USB is not my strong skill so I took low road and installed Ubuntu VM. For what is’s worth I got the same kind of error on Ubuntu.

Next step was to use official update procedure described in Since I wasn’t going to boot Linux on the board I didn’t need sample rootfs. So the whole procedure was:

- Go to L4T R21.4 page
- Download Tegra124_Linux_R21.4.0_armhf.tbz2
- Unpack it
- Connect microUSB port on device to Linux VM
- Get device into recover mode: power cycle, press and hold recovery button, press and release power button, release recovery button
- Run ./ jetson-tk1 mmcblk0p1, this should rewrite eMMC flash on the board and after reboot you will get u-boot prompt on serial console


At this point you can boot FreeBSD on TK1. I use netboot for most of my device so in this case it was: build and deploy world to /src/FreeBSD/tftproot/tk1, build and install kernel to the same directory, copy /src/FreeBSD/tftproot/tk1/boot/kernel/kernel to kernel.TK1 in tftproot directory, add entry do DHCP config and restart DHCP server. Entry looks like this:

host tk1 {
        hardware ethernet 00:04:4b:49:08:9e;
        filename "kernel.TK1";
        option root-path "/src/FreeBSD/tftproot/tk1";
        option root-opts "nolockd";
        option routers;

And also you need to add this to sys/arm/conf/JETSON-TK1 before building kernel:

options        BOOTP
options        BOOTP_NFSROOT
options        BOOTP_COMPAT
options        BOOTP_NFSV3

On the device you just run “dhcp; bootelf” and voila – it just works.


Next step was to get ubldr running. I prefer suing ubldr because it gives more control over boot process accessible from booted FreeBSD system. ubldr requires U-Boot with API support, so I had to rebuild U-Boot from sources provided by nvidia with added #define CONFIG_API and all standard patches from sysutils/u-boot-* ports. Build procedure is standard:

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make jetson-tk1_config

It will generate multiple files, u-boot-dtb-tegra.bin is the one you want.

To reflash board with non-standard u-boot run ./ -L /path/to/u-boot-dtb-tegra.bin jetson-tk1 mmcblk0p1

Back to ubldr. It was easy to build and load it. Build script:

export TARGET=arm
export TARGET_ARCH=armv6
export SRCROOT=/src/FreeBSD/wip
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj
export MAKESYSPATH=$SRCROOT/share/mk

set -x
set -e

buildenv=`make -C $SRCROOT TARGET_ARCH=armv6 buildenvvars`
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH obj
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH clean
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH UBLDR_LOADADDR=0x80600000 all

sudo cp /src/FreeBSD/obj/arm.armv6/src/FreeBSD/wip/sys/boot/arm/uboot/ubldr /src/FreeBSD/tftpboot/ubldr.TK1

Obviously, kernel.TK1 in DHCP config needs to be replaced with ubldr.TK1. 0×80600000 is some value I came up with by looking at u-boot default environment. Something not high enough to overlap with kernel and not low enough to overlap with u-boot.

And that’s where thing got hairy. To load ubldr and then netboot kernel, you need to set u-boot env loaderdev variable first: setenv loaderdev net; saveenv. And then do the same thing as above: dhcp; bootelf. Unfortunately I got this:

## Starting application at 0x81000098 ...
Consoles: U-Boot console
Compatible U-Boot API signature found @0xffa3e410

FreeBSD/armv6 U-Boot loader, Revision 1.2
(, Mon Jun 27 19:59:22 PDT 2016)

DRAM: 2048MB
MMC: no card present
MMC Device 2 not found
MMC Device 3 not found
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC Device 2 not found
Number of U-Boot devices: 3
U-Boot env: loaderdev='net'
Found U-Boot device: disk
Found U-Boot device: net
Booting from net0:
panic: arp: no response for

--> Press a key on the console to reboot <--
resetting ...

After some heavy thinking and code digging problem was narrowed down to u-boot network driver drivers/net/rtl8169.c. Instead of returning 0 on success and negative value on error it returns number of bytes sent on success and zero on error. Which confused ubldr into thinking nothing is sent, so recv part of exchange was never invoked. After fixing this issue kernel was loaded just fine but hang right afert

Using DTB compiled into kernel.
Kernel entry at 0x0x80800100...
Kernel args: (null)

Logn story short - it was caused by enabled D-Cache so I had to add


to u-boot config and go through rebuild/reflash cycle again. After this whole boot chain went through right to login prompt.

My next goal is to make TK1 self-contained box: get base system installed on eMMC and use attached SSD as scratch disk for swap and builds.

Few weeks back Ralf Nolden, who is *BSD champion in Qt community, urged me to clean-up and submit my Qt5-related projects to upstream and scfb platform plugin was picked as a test dummy. It took 12 iterations to get things right, along the way plugin was renamed to bsdfb, but eventually patch has been merged.

Next two candidates are bsdkeyboard and bsdsysmouse input plugins.

Two months ago I tried to setup dev environment using FreeBSD Vagrant box just to find out that FreeBSD does not support VirtualBox shared folders. After some googling I found Li-Wen Hsu’s github repository with some work in this area. Li-Wen and Will Andrews has already done major chunk of work: patches to VirtualBox build system, skeleton VFS driver, API to talk to hypervisor but hit a block with some implementation details in VirtualBox’s virtual-memory compatibility layer. Will provided very comprehensive analysis of the problem.

Li-Wen was occupied with some other projects so he gave me his OK to work on shared folder support on my own. Will’s suggestion was easy to implement – lock only userland memory, like Solaris driver does. VFS part was more complicated though: fs nodes, vnode, their lifecycle and locking is too hairy for drive-by hacking. I used tmpfs as a reference to learn some VFS magic, but a lot of things are still obscure. Nevertheless after few weeks of tinkering first milestone has been achieved: I can mount/unmount shared VirtualBox folder and navigate mounted filesystem without immediate kernel panic. Next goal (if time permits): stable and non-leaking read-only filesystem.

To those who do not track FreeBSD commit messages: I committed gpiokeys driver to -CURRENT as r299475. The driver is not enabled in any of the kernels but can be built as a loadable module.

For now it stays disconnected from main build because it breaks some MIPS kernel configs. Configs in question include “modules/gpio” as part of MODULES_OVERRIDE variable and since gpiokeys can be built only with FDT-enabled kernel the build fails.

gpiokeys can be used as a base for more input device driver: “gpio-keys-polled” and “gpio-matrix-keypad“. I do not have hardware to test this at the moment. If you do and you’re looking for small FreeBSD project to work on – here you go.

Next step on my ToDo list is to try tricking people into committing evdev patch, which at the moment is the only requirement for unlocking touchscreen support.

Qt5 for FreeBSD/Pi

January 14, 2015 — 1 Comment

Build SD card image using crochet-freebsd with option VideoCore enabled. Mount either SD card itself of image to build host

mount /dev/mmcsd0s2a /pi

Checkout Qt5 sources and patch them

cd /src
git clone git:// qt5
cd qt5
git checkout 5.4.0
./init-repository --module-subset=$MODULES

fetch -q -o - | patch -p1

Configure, build and install Qt5 to SD card

./configure -platform unsupported/freebsd-clang -no-openssl -opengl es2 -device freebsd-rasp-pi-clang -device-option CROSS_COMPILE=/usr/armv6-freebsd/usr/bin/ -sysroot /pi/ -no-gcc-sysroot -opensource -confirm-license -optimized-qmake -release -prefix /usr/local/Qt5 -no-pch -nomake tests -nomake examples -plugin-sql-sqlite

gmake -j `sysctl -n hw.ncpu`
sudo gmake install

You need BSD-specific plugins to enable mouse and keyboard input in EGLFS mode

cd /src/
git clone
cd qt5-bsd-input
sudo gmake install

Build application you’d like run and install it. I use one of the examples here

cd /src/qt5/qtbase/examples/opengl/cube
sudo gmake install

Unmount SD card, boot Pi, make sure vchiq is loaded

root@raspberry-pi:~ # kldload

Start application

root@raspberry-pi:~ # /usr/local/Qt5/examples/opengl/cube/cube -plugin bsdkeyboard -plugin bsdsysmouse

If you see something like this:

EGL Error : Could not create the egl surface: error = 0x3003

Or this:

QOpenGLFramebufferObject: Framebuffer incomplete attachment.

It means you need to increase GPU memory by setting gpu_mem in config.txt. Amount depends on framebuffer resolution. 128Mb works for me on 1920×1080 display.

bsdsysmouse plugin uses /dev/sysmouse by default, so you either should have moused running or specify actual mouse device, e.g.:

root@raspberry-pi:~ # cube -plugin bsdkeyboard -plugin bsdsysmouse:/dev/ums0

bsdkeyboard uses STDIN as input device, so if you’re trying to start app from serial console it should be something like this:

root@raspberry-pi:~ # cube -plugin bsdkeyboard -plugin bsdsysmouse < /dev/ttyv0

Audio on Raspberry Pi

January 9, 2015 — Leave a comment

With stable VCHIQ driver next obvious target was to add VCHIQ-based audio support. So let me introduce to you: vchiq_audio, first take. It’s part of vchiq-freebsd repo so if you use Crochet to build SD card image just enable option VideoCore in config file and module will be automatically included.

From shell run kldload vchiq_audio and you’re good to do. I believe that audio output is picked up automatically by VideoCore so if you have HDMI connected it’s probably going to be HDMI. I do not have device to confirm this. Adding knob to control audio output (auto, headphones, HDMI) is on my ToDo list.

Quality is not ideal though. From quick tests it seems to work fine on system with rootfs on NFS but there are audio drops on SD-based system while playing mp3 over NFS. I’m going to debug and stresstest it more thoroughly next week.

Short instruction on how to install mpg321 package on RPi:

env PACKAGESITE= SIGNATURE_TYPE=none pkg bootstrap

mkdir -p /usr/local/etc/pkg/repos
cd /usr/local/etc/pkg/repos
echo 'FreeBSD: { enabled: no }' > FreeBSD.conf

cat > chips.ysv.conf <<__EOF__
chips.ysv: {
  url: "",
  mirror_type: "http",
  signature_type: "none",
  enabled: yes

pkg install mpg321

Update: support for keyboard/mouse has been added

After New Year I got back to hacking the VCHIQ stuff (thanks to adrian@ for prodding). Since last time I touched NetBSD folks got it merged to main tree, syncing with latest upstream code and fixing some stupid bugs in my codebase. So I partially merged things back, spent some time on fixing more bugs introduced by yours truly, merged userland bits from latest Broadcom’s bits (and fixing some bugs introduced by them). And as a result VCHIQ got stable enough to run ioquake3d on raspberry pi. Well, you can’t play it because there is no sound and no mouse support and keyboard support is severely crippled but you can navigate menus and watch demoes.

Here is short summary of how to get it running:

  • Get latest HEAD that includes r276794
  • Get latest crochet-freebsd
  • Create configuration file for RasspberryPi, make sure that it’s configured for 2Gb SD card and has VideoCore enabled. i.e. it contains:
    option ImageSize 1950mb # for 2 Gigabyte card
    option VideoCore
  • Build RPi image and flash to SD card
  • mount FreeBSD partition, e.g. mount /dev/mmcsd0s2a /mnt
  • Copy Quake3 PAK files to /baseq3 directory on SD card
  • Download and copy *.so files to /baseq3 and ioqake3.arm to /usr/bin on SD card
  • Unmount FreeBSD partition and mount boot partiotion, e.g. mount_msdosfs /dev/mmcsd0s1 /mnt
  • Edit config.txt and change gpu_mem value to 64
  • Unmount SD card and boot it on your Pi
  • Load vchiq module: kldload vchiq
  • Start Quake3: ioqake3.arm +set s_initsound 0

Keyboard support is really broken. TAB and ENTER works, so you can navigate menus. But that’s pretty much it.

ioquake3 codebase with my minor changes located here:
I provide pre-compiled binaries because for some reason ioquake3 built with xdev tools crash in qsort (libc incompatibilities?) so I use make buildenv to build it.

And here is photo of demo in action (there are RaspberryPi and ZedBoard on it too, yay!)
Photo Jan 07