I finally got around to BSDify my Jetson TK1. Here is short summary of what is involved. And to save you some scrolling here are artifacts obtained from whole ordeal: https://people.freebsd.org/~gonzo/arm/jetson-tk1/

Jetson TK1


First of all – my TK1 didn’t have U-Boot. Type of bootloader depends on the version of Linux4Tegra TK1 comes with. Mine had L4T R19, with some kind of “not u-boot” bootloader. My first attempt was to use tegrarcm tool, it uses libusb, so it’s possible to build it on FreeBSD with some elbow grease, but once I tried to run it – it gave me cryptic errors and USB is not my strong skill so I took low road and installed Ubuntu VM. For what is’s worth I got the same kind of error on Ubuntu.

Next step was to use official update procedure described in http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/l4t_quick_start_guide.txt. Since I wasn’t going to boot Linux on the board I didn’t need sample rootfs. So the whole procedure was:

- Go to L4T R21.4 page
- Download Tegra124_Linux_R21.4.0_armhf.tbz2
- Unpack it
- Connect microUSB port on device to Linux VM
- Get device into recover mode: power cycle, press and hold recovery button, press and release power button, release recovery button
- Run ./flash.sh jetson-tk1 mmcblk0p1, this should rewrite eMMC flash on the board and after reboot you will get u-boot prompt on serial console


At this point you can boot FreeBSD on TK1. I use netboot for most of my device so in this case it was: build and deploy world to /src/FreeBSD/tftproot/tk1, build and install kernel to the same directory, copy /src/FreeBSD/tftproot/tk1/boot/kernel/kernel to kernel.TK1 in tftproot directory, add entry do DHCP config and restart DHCP server. Entry looks like this:

host tk1 {
        hardware ethernet 00:04:4b:49:08:9e;
        filename "kernel.TK1";
        option root-path "/src/FreeBSD/tftproot/tk1";
        option root-opts "nolockd";
        option routers;

And also you need to add this to sys/arm/conf/JETSON-TK1 before building kernel:

options        BOOTP
options        BOOTP_NFSROOT
options        BOOTP_COMPAT
options        BOOTP_NFSV3

On the device you just run “dhcp; bootelf” and voila – it just works.


Next step was to get ubldr running. I prefer suing ubldr because it gives more control over boot process accessible from booted FreeBSD system. ubldr requires U-Boot with API support, so I had to rebuild U-Boot from sources provided by nvidia with added #define CONFIG_API and all standard patches from sysutils/u-boot-* ports. Build procedure is standard:

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make jetson-tk1_config

It will generate multiple files, u-boot-dtb-tegra.bin is the one you want.

To reflash board with non-standard u-boot run ./flash.sh -L /path/to/u-boot-dtb-tegra.bin jetson-tk1 mmcblk0p1

Back to ubldr. It was easy to build and load it. Build script:

export TARGET=arm
export TARGET_ARCH=armv6
export SRCROOT=/src/FreeBSD/wip
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj
export MAKESYSPATH=$SRCROOT/share/mk

set -x
set -e

buildenv=`make -C $SRCROOT TARGET_ARCH=armv6 buildenvvars`
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH obj
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH clean
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH UBLDR_LOADADDR=0x80600000 all

sudo cp /src/FreeBSD/obj/arm.armv6/src/FreeBSD/wip/sys/boot/arm/uboot/ubldr /src/FreeBSD/tftpboot/ubldr.TK1

Obviously, kernel.TK1 in DHCP config needs to be replaced with ubldr.TK1. 0×80600000 is some value I came up with by looking at u-boot default environment. Something not high enough to overlap with kernel and not low enough to overlap with u-boot.

And that’s where thing got hairy. To load ubldr and then netboot kernel, you need to set u-boot env loaderdev variable first: setenv loaderdev net; saveenv. And then do the same thing as above: dhcp; bootelf. Unfortunately I got this:

## Starting application at 0x81000098 ...
Consoles: U-Boot console
Compatible U-Boot API signature found @0xffa3e410

FreeBSD/armv6 U-Boot loader, Revision 1.2
(gonzo@eb3.bluezbox.com, Mon Jun 27 19:59:22 PDT 2016)

DRAM: 2048MB
MMC: no card present
MMC Device 2 not found
MMC Device 3 not found
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC Device 2 not found
Number of U-Boot devices: 3
U-Boot env: loaderdev='net'
Found U-Boot device: disk
Found U-Boot device: net
Booting from net0:
panic: arp: no response for

--> Press a key on the console to reboot <--
resetting ...

After some heavy thinking and code digging problem was narrowed down to u-boot network driver drivers/net/rtl8169.c. Instead of returning 0 on success and negative value on error it returns number of bytes sent on success and zero on error. Which confused ubldr into thinking nothing is sent, so recv part of exchange was never invoked. After fixing this issue kernel was loaded just fine but hang right afert

Using DTB compiled into kernel.
Kernel entry at 0x0x80800100...
Kernel args: (null)

Logn story short - it was caused by enabled D-Cache so I had to add


to u-boot config and go through rebuild/reflash cycle again. After this whole boot chain went through right to login prompt.

My next goal is to make TK1 self-contained box: get base system installed on eMMC and use attached SSD as scratch disk for swap and builds.

Few weeks back Ralf Nolden, who is *BSD champion in Qt community, urged me to clean-up and submit my Qt5-related projects to upstream and scfb platform plugin was picked as a test dummy. It took 12 iterations to get things right, along the way plugin was renamed to bsdfb, but eventually patch has been merged.

Next two candidates are bsdkeyboard and bsdsysmouse input plugins.

Two months ago I tried to setup dev environment using FreeBSD Vagrant box just to find out that FreeBSD does not support VirtualBox shared folders. After some googling I found Li-Wen Hsu’s github repository with some work in this area. Li-Wen and Will Andrews has already done major chunk of work: patches to VirtualBox build system, skeleton VFS driver, API to talk to hypervisor but hit a block with some implementation details in VirtualBox’s virtual-memory compatibility layer. Will provided very comprehensive analysis of the problem.

Li-Wen was occupied with some other projects so he gave me his OK to work on shared folder support on my own. Will’s suggestion was easy to implement – lock only userland memory, like Solaris driver does. VFS part was more complicated though: fs nodes, vnode, their lifecycle and locking is too hairy for drive-by hacking. I used tmpfs as a reference to learn some VFS magic, but a lot of things are still obscure. Nevertheless after few weeks of tinkering first milestone has been achieved: I can mount/unmount shared VirtualBox folder and navigate mounted filesystem without immediate kernel panic. Next goal (if time permits): stable and non-leaking read-only filesystem.

To those who do not track FreeBSD commit messages: I committed gpiokeys driver to -CURRENT as r299475. The driver is not enabled in any of the kernels but can be built as a loadable module.

For now it stays disconnected from main build because it breaks some MIPS kernel configs. Configs in question include “modules/gpio” as part of MODULES_OVERRIDE variable and since gpiokeys can be built only with FDT-enabled kernel the build fails.

gpiokeys can be used as a base for more input device driver: “gpio-keys-polled” and “gpio-matrix-keypad“. I do not have hardware to test this at the moment. If you do and you’re looking for small FreeBSD project to work on – here you go.

Next step on my ToDo list is to try tricking people into committing evdev patch, which at the moment is the only requirement for unlocking touchscreen support.

Qt 5.6 is finally out so I thought I’d give it a spin on my Raspberry Pi. Previously I used cross-compilation but this time I thought I’d spend some time in trying to create ports Qt modules. There is Qt 5.5.1 in ports and it’s nicely split into sub-ports and most of gory details are hidden in bsd.qt.mk library. The problem with it is it’s highly coupled with Xorg stuff and I didn’t find easy way to squeeze non-desktop use cases into current infrastructure. So I just created new custom devel/qt56 port.

In order to get it done as fast as possible I took several shortcuts: all the stuff is installed to /usr/local/qt5 directory, there is no meta-port for submodules to share common part yet. Also besides base the only module I packaged (I was particularly interested in it) was QtMultimedia. Should be fairly easy to fix last 2 items though.

Qt layer for UI provider is called QPA: Qt Platform Abstraction. There are quite a few of them but I am familiar and interested in two: plain framebuffer and eglfs. Plain framebuffer stock QPA plugin is called linuxfb and naturally we can’t use it for FreeBSD. Luckily there is a lot of similarities between Linux fb and syscons(or vt) fb (you can’t get very innovative with framebuffer) so writing QPA support for scfb was easy. It can be used with any generic SoC with framebuffer support: AM335x(beaglebone black), i.MX6(Wandboard), NVIDIA Tegra, Pi.

elgfs is full-screen OpenGL mode. OpenGL implementation depends on SoC vendor, e.g. Pi’s library is provided by raspberrypi-userland port. If we had OpenGL support for AM335x it would have been provided by PowerVR userland libraries. As far as I understand eglfs can’t be made universal: each implementation has its own quirks so you have to specify target OpenGL vendor during build time. So far we support eglfs only for Raspberry Pi thanks to Broadcom’s open-sourcing kernel drivers and userland libraries.

Then there is question of input. When you start your application from console you have several options to get user’s input: keyboard, mouse, touchscreen. Common way to do this on Linux is through evdev which is a universal ways to access these kinds of devices. There is effort to get this functionality on FreeBSD so when it’s there stock input plugins could be used as-is. Until then there are two non-standard plugins by yours truly: bsdkeyboard and bsdsysmouse.

scfb, bsdysmouse, and bsdkeyboard are included in experimental ports as patches.

All in all experience of getting Qt 5.6 running on FreeBSD/Pi was smooth. And to make this post more entertining here are two demos running on my Pi2 with official touchscreen.

Qt demo player with visualizer (src):

OpenGL demo with Qt logo in it (src):

Being able to power cycle ARM boards remotely (without spending a lot of $$$) was on my wish list for way to long, so I finally got around to put something together. The obvious way to do this is power relay controlled by GPIO + remotely accessible GPIO port. For the former I picked up this relay by Digital Loggers. It was four ports, only two of them are connected at a time, you can switch selected pair by setting control port level. For a controller part I picked up RIoTBoard but actually any ARM board with sshd running on it and user-accessible GPIO pin would do.

On riotboard there are several GPIO pins available on J13 expansion port. You can find values in board’s user manual. Just in case you don’t know (and it’s not in manual) pin 1 on the port marked with white triangle, if you look at port so that pin 1 is in top-left corner, next to it on the right is pin 2, and right under it is pin 3. So it goes like:
[ 1 2 ]
[ 3 4 ]
[ 5 6 ]
… and so on …

I used GPIO4_16 (pin 5). You need to connect GND and GPIO pin to relay’s control port. Polarity does not matter in this case. For port control you can use gpioctl utility. But first we need to identify this pin on FreeBSD’s side. For each GPIO bank FreeBSD has /dev/gpiocX device node, where X starts from zero. But it’s not uncommon for SoC vendors to start numbering GPIO banks from 1. riotboard’s vendor is one of them so GPIO4 is actually controlled through /dev/gpioc3. By default pin 16 is configured as input, so you need to change it to out:

# gpioctl -f /dev/gpioc3 -c 16 OUT

and then you can start giving orders to relay:

# gpioctl -f /dev/gpioc3 16 0
# gpioctl -f /dev/gpioc3 16 1
# gpioctl -f /dev/gpioc3 -t 16

Demo video:

Back from vacation and back to work. Once I got RIoTboard up and running next natural step was to wrap up some i.MX6 project I had in “almost finished” state for months. So now they’re in “going through review” state: drivers for HDMI framer and IPU. They add basic 1024×768 console for iMX6 board. Video mode management requires more sophisticated timers framework, that is being work on as a part of Jetson TK1 port.

RIoTboard support

November 12, 2015 — Leave a comment

My career as a trendy videoblogger starts to pay off. Nice people from Newark element14 offered to send me some hardware for experiments (no strings attached) I took them up on their offer and few days later received RIoTboard. It’s iMX6 Solo in developer-friendly package, not as compact as Beaglebone but nicely built and comes with more connectors.

FreeBSD’s iMX6 support is very good, so it took two one-line fixes to FreeBSD kernel code to make it work on RIoTboard. The other chunk of work was U-Boot package. Took more time than it should have due to some operator errors. The bring up process is more or less the same as for any other iMX6 system, so it should be really easy to add this board to crochet. Step by step it looks like this (some of the code came from crochet):

Prepare environment for the build

export TARGET=arm
export TARGET_ARCH=armv6
export SRCROOT=/src/FreeBSD/head
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj
export MAKESYSPATH=$SRCROOT/share/mk

Build world and kernel

make -j16 -C $SRCROOT buildworld
make -C $SRCROOT KERNCONF=$KERNCONF -j16 buildkernel

Create and partition SD card image


rm -f $IMG
dd if=/dev/zero of=$IMG bs=1000000 count=1024
MDUNIT=$(mdconfig -a -f $IMG)
gpart create -s mbr $DEV

# Create FAT partition
gpart add -a 63 -b 16384 -s 50m -t '!12' $DEV
gpart set -a active -i 1 $DEV
newfs_msdos -L "BOOT" -F 16 ${FAT_DEV}

# Create UFS partition
gpart add -t freebsd  $DEV
gpart create -s BSD $UFS_DEV
gpart add -t freebsd-ufs -a 64k $UFS_DEV

newfs $UFS_PART
# Turn on Softupdates
tunefs -n enable $UFS_PART
tunefs -j enable -S 4194304 $UFS_PART
# Turn on NFSv4 ACLs
tunefs -N enable $UFS_PART

mdconfig -d -u $MDUNIT

Mount UFS partition

MDUNIT=$(sudo mdconfig -a -f $IMG)

sudo mount $UFS_DEV $MNTDIR

Install world + kernel

sudo -E make -C $SRCROOT installworld -DDB_FROM_SRC DESTDIR=$MNTDIR
sudo -E make -C $SRCROOT distribution -DDB_FROM_SRC DESTDIR=$MNTDIR
sudo -E make -C $SRCROOT installkernel DESTDIR=$MNTDIR

Install come configuration. We need hw.fdt.console in loader.conf because riotboard’s DTS file does not have stdout-path property in chosen node. In this case FreeBSD falls back to serial0 node but user-accessible UART on RIoTboard is UART2 so as a result no kernel output visible on serial port.

echo 'hw.fdt.console="/soc/aips-bus@02100000/serial@021e8000"' > /tmp/loader.conf
cat > /tmp/fstab <<__EOF__
/dev/mmcsd0s2a  /               ufs rw,noatime          1 1
/dev/mmcsd0s1   /boot/msdos     msdosfs rw,noatime      0 0
tmpfs           /tmp            tmpfs rw,size=31457280  0 0
tmpfs           /var/log        tmpfs rw,,size=15728640 0 0
tmpfs           /var/tmp        tmpfs rw,size=5242880   0 0
cat > /tmp/rc.conf <<__EOF__

# minimal network config

# turn off sendmail

sudo mv /tmp/loader.conf $MNTDIR/boot/loader.conf
sudo mv /tmp/fstab $MNTDIR/etc/fstab
sudo mv /tmp/rc.conf $MNTDIR/etc/rc.conf
sudo mkdir $MNTDIR/boot/msdos

Unmount UFS and mount FAT partition

sudo umount $MNTDIR

sudo mount -t msdosfs $FAT_DEV $MNTDIR

Build and install ubldr, unmount FAT partition

buildenv=`make -C $SRCROOT TARGET_ARCH=armv6 buildenvvars`
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH obj
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH clean
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH depend
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH UBLDR_LOADADDR=0x12000000 all

sudo cp ${MAKEOBJDIRPREFIX}/arm.armv6/${SRCROOT}/sys/boot/arm/uboot/ubldr $MNTDIR
sudo umount $MNTDIR

Download and install u-boot-riotboard port. Then flash u-boot to the image

sudo dd if=/usr/local/share/u-boot/u-boot-riotboard/u-boot.imx of=$DEV bs=1k oseek=1 conv=sync

Detach md device

sudo mdconfig -d -u $MDUNIT

Configure boot switch selector on board to boot from SD card as described in this document

Insert SD card to either SD slot or uSD slot. When powered up you should see u-boot prompt


Boot command would look something like:

=> fatload mmc 0 $loadaddr ubldr
=> bootelf

For SD slot mmc unit is 0, for uSD it's probably 1, I didn't test.

Boot log: here

Next step for me is to get my imx6_video branch up to date and see if I can commit it to HEAD.

FreeBSD on RPi 2 progress

November 9, 2015 — 4 Comments

Quick update on progress in FreeBSD’s support of Raspberry Pi:
- VCHIQ driver was updated to the latest vendor code and bunch of FreeBSD-specific problems was fixed: locking, handling of non-cacheline aligned data
- Raspberry Pi userland code was updated to the latest vendor code
- Mikael Urankar created misc/raspberrypi-userland for userland libraries/utilities
- Mikael also created multimedia/omxplayer port for OMXPlayer, video player developed for RPi. It’s also used in Kodi player on Pi.
- I created misc/ioquake-pi port to make it easier for people to try it out

I put together all this stuff on my brand new Pi 2 and recorded demo that showcases, OpenGL, Quake3, omxplayer, camera, and audio. I hope all these ports will be committed/updated before next round of armv6 packages build.

Received yesterday and had to assembly it first thing today. Display part works like a charm without any system modifications. Haven’t researched touchscreen part though.