Archives For FreeBSD

Last year I got my hands on Minnowboard Turbot (courtesy of Frank H.) and spent some time working on communications protocols support for it. Below is short summary of what works and what doesn’t.

Minnowboard Turbot is Atom-base SoC, and standard x86 part (HDMI, network, USB) FreeBSD just works on it. The board has expansion connector that exposes I2C, SPI, and GPIO pins and can be used to talk to peripheral devices. Short summary of header pins can be found on


GPIO functionality on Minnowboard provided by bytgpio(4) driver (name was chosen to match OpenBSD one). There are three banks, but the most accessible is the third one available via /dev/gpioc2. All GPIO pins on the expansion header are pin number in this bank. So to set pin “GPIO 0″ on the picture to HIGH you would use gpioc -f /dev/gpioc2 0 1.

To use some GPIO peripheral that has FreeBSD kernel driver you will have to describe it using hints in /boot/loader.conf. For instance there is LED D2 that is attached to pin 22. So to expose it through gpioled(4) interface you’d add following lines to /boot/loader.conf:

# pin 22: (1 << 22)


Driver for SPI support on Minnowboard is intelspi. Hardware supports all four SPI modes and clock frequency configuration but the driver doesn’t (yet). So it’s always mode 0 and frequency is hardcoded to 10MHz. To access it from userland you’d use spigen driver for now. As with gpio peripherlas, you’d have to add spigen via hints:


There is no spigen module, so it has to be enabled in kernel config:

device          spibus
device          spigen


I2C driver is ig4(4). There are two I2C buses, the one that is accessible from expansion header is iicbus0. As with previous two protocols devices should be defined by hints. For example:

# 0x68 in 8-bit format
# for example only, hasn't been tested

There is no ds3231 module so again, you need to enable it in kernel:

device          iicbus
device          ds3231

To use I2C from userland via /dev/iicN you’ll need to load iic(4) module.

Over years I accumulated fair number of devices I have no real use for. I ordered them either on impulse, or to add couple of $$$ to the bill to get free delivery. So in order to get at least some value from those purchases I put together goofy demo using two of such devices: I2C temperature sensor breakout from Sparkfun and 128×32 SPI OLED display from Adafruit.

I wrote two libraries to talk to TMP102 (chip in which temp sensor is based) over I2C and to SSD1306(OLED display chip) over SPI and several demos simple enough to put together in one day but flashy enough to excite my inner child and bring fond memories of MSDOS days. The SPI chip requires this fix in kernel. Userland SPI API provides only very basic functionality but luckily it was enough to talk to SSD1306.

TMP102 is I2C-only device but SSD1306 uses two signals in addition to standard SPI ones and GND/VCC: Data/Command switch and Reset. I connected them to GPIO pins 23 and 24 on my RPi. SSD1306 code is based on Adafruit’s Python library and supports only one model so far, but can easily be extended to support all three of them.


rfkill on Jetson TK1

December 11, 2016 — Leave a comment

If you’re trying to install half-sized mini-PCIe wifi card in Jetson TK1, be aware, that the board has rfkill feature that is enabled by default. rfkill is hardware or software controlled switch that enables/disables RF signal on the wifi card itself. In case of mini-PCIe it’s controlled by level on pin 20 of the card. On Jetson TK-1 that pin is connected to GPIO X.7 pin. So to enable wifi on the board you need to run something like:

gpioctl -cN gpio_X.7 OUT
gpioctl -N gpio_X.7 1

Update (11/29/2016): DTB overlay no longer required. PSCI monitor will patch in-memory DTB to add psci node. Commit

Short version of how to get RPi3 with SMP support

Long version of what’s in those .bin/.dtbo files

Boot sequence for PSCI monitor on RPi3 looks like this:

  • entry point is called at EL3
  • monitor performs some CPU-specific initializations
  • monitor sets up exceptions vector with SMC handler
  • monitor reserves memory so it’s not going to be overwritten by next boot stage
  • monitor drops to EL2, passes control to next boot stage

First problem arises at step #1. If you generate binary and boot it using kernel parameter in config.txt by the time control is passed to entry point CPU is already at EL2. It’s because VideoCore firmware does not pass control directly to kernel but runs some built-in code prior to that which is called ARM stub. Good news – that built-in code can be overridden. The history behind ARM stubs can be found in this thread. And source code for default ARMv8 stub is in armstub8.S file.
It can be overridden either by adding armstub=NNN.bin line to config.txt or by copying stub to armstub8.bin file on FAT partition, this file is loaded automatically if it’s present.

Default stub performs some initialization and runs following pseudocode on each CPU:

void *spin_entry[4];
if (current_cpu == 0)
else {
    while (spin_entry[current_cpu] == 0) {

It’s a good starting point so I added exception vector that jumps to PSCI handler routine, and PSCI entry points for PSCI_VERSION, CPU_ON, SYSTEM_REBOOT, SYSTEM_OFF, SYSTEM_RESET.

So far so good. Back to memory reservation. My initial idea was for PSCI to edit DTB blob. Either add one more memreserve block that starts at 0×0 and 16Kb long or edit /memory node and cut 16Kb from first reg pair. I ended up implementing both of these approaches and neither worked. Because on RPi3 boot sequence looks like: u-boot -> loader.efi -> kernel. All memory information is passed to kernel via EFI and this part of DTB is just ignored. And U-Boot does not care about FDT either. It gets available memory size by querying VideoCore directly using mailbox API and then initializes on DRAM bank: [0..dram_size].

After short while of heavy thinking I ended up with just passing amount of reserved memory as the second argument of U-Boots entry function and then using that amount to start DRAM bank range with. Hackish but works.

Now there is a matter of VideoCore-loaded FDT blob. Most of the FreeBSD images out there have device_tree_address=0×100 line in config.txt. This is reasonable value so FDT blob does not overlap with u-boot. Unfortunately new ARM stub with PSCI functionality is larger than 0×100 bytes and now overlaps with it. So device_tree_address has to be bumped to 0×4000. But U-Boot has 0×100 hardcoded as default so config.txt and u-boot now out of sync. On the other hand ARM stub (default and PSCI-enabled) does pass FDT blob address as the first argument to entry function. So u-boot can set value of fdt_addr_r to correct address dynamically.

PSCI: check, U-Boot: check. There is one more missing part. FreeBSD kernel identifies presence of PSCI by checking for FDT node with “arm,psci-0.2″ compatibility string. Obviously there is no such thing in original dtb file. PSCI monitor adds this node dynamically by patching DTB blob in-memory before passing control to U-Boot. It would be natural to make PSCI monitor patch FDT blob and add this new node but I wasn’t up to the task of implementing it in assembler. I took the low road and just created DTB overlay with psci node. That’s why psci.dtbo and “dtoverlay=psci” line in config.txt are required. I’m planning to add C files to PSCI monitor and implement dynamic DTB patching but for now overlay should do.

Source code for all the stuff in this post: rpi3-psci-monitor and u-boot patches.

Precompiled bits: rpi3-smp

Getting SMP support for RPI3 took some time and was interesting learning experience. So I share bits of what I learned here.

FreeBSD/arm boot start on one CPU (called primary) the rest of CPUs are “on hold”. At some point in boot sequence non-primary CPUs are forced to call mpentry() function. The way CPU is forced to call mpentry is platform-dependent. RPi2 for instance passes mpentry and argument to VideoCore using mailbox interface and then VideoCore kicks ARM CPU in action. So most of SoC has their own implementation of platform_mp_start_ap platform API method.

FreeBSD/arm64 takes more standardized approach. It relies on secure monitor to provide this functionality. To explain what secure monitor is I need to diverge a bit into Aarch64 architecture. Aarch64 has four Exception Levels: EL3, EL2, EL1, EL0. Higher level acts as a “kernel” to the lower level. Most usual roles for exception levels are:

EL3 – Secure Monitor
EL2 – Hypervisor
EL1 – OS kernel
EL0 – userland apps

Just like userland can request kernel service through syscal lower exception level can request service from higher one through syscall-like mechanism. On Aarch64 it’s SMC, HVC, SVC instruction to get to EL3, EL2, EL1 respectively. There are multiple services that EL1 can request from Hypervisor or Secure Monitor but one of them is PSCI (Power State Coordination Interface). It’s an API to control CPUs and power states of the system. Full spec can be found here.

SMP support on FreeBSD/arm64 requires Secure Monitor and there is no such thing for Raspberry Pi 3. Normally it’s a piece of software that is provided as a part of SoC. Stage 1 boot loader resides in secure ROM and every subsequent boot stage can be verified using certificates. There are regions of memory where secure monitor is loaded that can not be accessed from non-secure exception levels. No such thing on RPi3 either.

There were 3 options to get PSCI on RPi3: write “bare metal” implementation from the scratch, add it to U-Boot or port existing open source secure monitor to RPi3. Since U-Boot was already a part of the setup I looked up if there was PSCI support for it. There was for ARMv7 in the mainline and for AMRv8 as a patchset with generic PSCI framework and implementation for one of the platforms. Framework relied on the existence of secure memory region which is not a thing on RPi3 so I went for another option: port existing monitor.

Reference implementation of SM is ARM Trusted Firmware. It’s very well documented and easily extendable but the problem was that just like with U-Boot it made assumptions about memory layout that were not true for RPi3 and were not easy to change in the ATF code.

At this point bare metal PSCI emulation looked like less work than the other two options and I started experimenting with really low level stuff. I will write my progress up in the next blog post.

MFC tracking web-tool

October 16, 2016 — Leave a comment

Next week I plan to do bunch of MFCs for ARM and evdev stuff and to make it less of an ordeal I made this tool to track what’s merged and what’s not: It provides basic functionality I thought I would need for this process:

  • Navigator for HEAD commits with visual representation of MFC state: “no mfc scheduled”, “ready for mfc”, “waiting for ‘mfc after’ date”, “merged”
  • Basic filtering: by author name, by two states: “waiting”, and “ready”
  • “MFC basket” – manage set of commits I would like to merge back in one go
  • Generate svn command and change log for selected “MFC basket”

Stack used: Django + Bootstrap3 + jQuery

GitHub project: mfctracker

Raspberry Pi 3 limited support was committed to HEAD. Most of drivers should work with upstream dtb, RNG requires attention because callout mode seems to be broken and there is no IRQ in upstream device tree file. SMP is work in progress. There are some compatibility issue with VCHIQ driver due to some assumptions that are true only for ARM platform.

SD card layout is the same as for RPi and RPi2 but boot chain is different. All ARM64 supported by FreeBSD up to now used EFI as boot environment. RPi 3 has only VC firmware and whatever it can spin off, e.g. u-boot. So it seemed easier to enable EFI API in U-Boot instead of porting ubldr to arm64. There were some hiccups with netbooting, (see patch) but otherwise it was OK. U-Boot port and crochet config for Pi 3 should be committed “real soon”(tm).

For those who would like to try it ASAP Shawn Webb put together instruction on how to get bootable SD image.

Short summary of couple of frustrating hours I spent trying to get my RPi3 netbooting:

U-Boot – 2016.09
raspberrypi/firmware/boot – ec63df146f454e8cab7080380f9138246d877013

armstub.bin, armstub7.bin, armstub8.bin – NOT REQUIRED. There are tens of google results mentioning these files along with some dd magic – they all obsolete. Aforementioned version of firmware does not require them. 64-bit mode is controlled by arm_control variable in config.txt (see below). If 64-bit mode requested default kernel name becomes kernel8.img, and kernel load address becomes 0×80000. U-Boot uses correct default load address so no need for any additional parameters or hacks.

Building u-boot:

# gmake ARCH=arm CROSS_COMPILE=aarch64-none-elf- CONFIG_EFI=y rpi_3_defconfig
# gmake ARCH=arm CROSS_COMPILE=aarch64-none-elf- CONFIG_EFI=y 

Boot partition:
- Copy all files/dirs from firmware/boot to your FAT partition (let’s say it’s mounted at /mnt/fat)
- Copy u-boot.bin to kernel8.img on your FAT partition
- Create config.txt with following content:


If you want to keep u-boot.bin name and not use kernel8.img, add following ling:


Unmount and boot RPi3. You should see U-Boot output on both serial console and HDMI.

Default U-Boot build uses mini-uart. So if for some reason you want to use PL011 instead, patch include/configs/rpi.h and make sure that CONFIG_BCM283X_MU_SERIAL is not defined and CONFIG_PL01X_SERIAL is defined instead.

config.txt in this case should look like:


Few weeks ago evdev support was finally committed to HEAD. Project started a part of SoC 2014 by Jakub Klama and then picked up, finished and submitted by Vladimir Kondratiev. It’s drop-in compatible with Linux API which means all you need to do is add #ifdef __FreeBSD_ around respective includes and existing code (if it’s otherwise cross-compatible with FreeBSD) should just work. Which is the case for Qt and to lesser extent for tslib. Hardware support is still moving target, FreeBSD has evdev-compatible drivers for USB keyboards, USB mice, TI’s AM33xx touchscreen controller and Raspberry Pi’s official touchscreen. Only the latter device supports multitouch and Vladimir submitted patch required to get it working. To my knowledge it’s the first multitouch touchscreen ever working on FreeBSD so I decided to record demo to save this moment for generations to come. Well, not really. Mostly to brag and to let people know that it’s possible and encourage them to make stuff and experiment with FreeBSD, ARM, and Qt.

Demo below is standard imagegestures example built using latest dev branch of Qt.

I spent Labor Day weekend laboring on VBox shared folders support for FreeBSD. It’s been some time since I worked on it last time so I had to refresh my memory first. Things have moved on since then – VBox in ports was updated to version 5, but fortunately Li-Wen synced up freebsd-vboxfs repo to the latest version. After three days of laid-back hacking I am glad to announce that following VOPs are kind of implemented (in no particular order): lookup, access, readdir, read, getattr, readlink, remove, rmdir, symlink, close, create, open, write. “Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host’s md5sum, create/remove directories, unzip zip file, etc but I doubt it would survive stress-test. Locking is all wrong at the moment and read/write VOPs allocate buffers for every operation.

I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them in vboxfs nodes, but if one of parent directories is renamed, all cached names should be modified accordingly. I am going to tackle these two problems once I have long enough stretch of time time sit and concentrate on task.