Archives For FreeBSD

I wrote an introductory article on how the audio subsystem on SBCs work: CODECs, I2S, DTS, whole nine yards. WordPress editor didn’t seem to be a very convenient tool for this kind of write up so I gave asciidoc a try and so far liked it.

Link to the article:

Some time ago Filippo Valsorda wrote yubikey-agent, seamless SSH agent for YubiKeys. I really like YubiKeys and worked on the FreeBSD support for U2F in Chromium and pyu2f, getting yubikey-agent ported looked like an interesting project. It took some hacking to make it work but overall it wasn’t hard. Following is the roadmap on how to get it set up on FreeBSD. The actual details depend on your system (as you will see)

The first step is to set up middleware for accessing smart cards (YubiKey implements CCID smart-card protocol). The pcsc-lite package provides a daemon and a library for clients to communicate with the daemon. ccid is a plugin for pcsc-lite that implements the actual CCID protocol over USB. devd rules are to make the daemon re-scan USB devices on hotplug

sudo pkg install ccid pcsc-lite
sudo mkdir -p /usr/local/etc/devd
sudo tee /usr/local/etc/devd/pcscd.conf << __EOF__
attach 100 {
        device-name "ugen[0-9]+";
        action "/usr/local/sbin/pcscd -H";
detach 100 {
        device-name "ugen[0-9]+";
        action "/usr/local/sbin/pcscd -H";

sudo service devd restart
sudo sysrc pcscd_enable="YES
sudo service pcscd start

go and git are build requirements for the app. go get command is required because FreeBSD support was only recently merged into piv-go and the latest release, referenced in go.mod, still does not have it. go install command installs the app in ~/go/bin/. When new version of piv-go is released and yubikey-agent switches to using it, all these commands can be replaced with single go get command.

sudo pkg install go git
git clone
cd yubikey-agent
go get
go build
go install

The binary is in ~/go/bin/ directory, you can add it to your $PATH to type less.

The next step is setting up a Yubikey, it’s well documented on the official site. One caveat though is if PIN length is less than 6 chars the setup fails with the somewhat confusing message: “‼️ The default PIN did not work.”

The actual usage of yubikey-agent depends on your setup. First of all, yubikey-agent is a “eventually GUI” app. At some point, when entering PIN is required it starts pinentry command which task is to present user with a dialog entry, get PIN, and pass it to the app. There are multiple pinentry flavors, with different front-ends: TTY, Qt, GTK. gopass module used in yubikey-agent does not work with plain TTY backend, and requires pinentry to be a GUI app. On Debian/Ubuntu users can switch between flavors and make /usr/bin/pinentry point to either Qt5 or GTK version, but in FreeBSD /usr/local/bin/pinentry is always TTY one. I worked around it by installing pinentry-qt5 package and making symlink.

sudo pkg install pinentry-qt5
sudo ln -s /usr/local/bin/pinentry-qt5 /usr/local/bin/pinentry

If you need /usr/local/bin/pinentry to be TTY version for some other stuff, there may be a problem. How to work around this depends on your requirements. I don’t have a ready recipe.

Because yubikey-agent is “eventually GUI” you can either start it in .xsession or .xinitrc files or start it some other way with DISPLAY env variable set. Other than that official documentation is a good source of information on how to use it.

Whenever I sit down to work at the coffee shop I’ve never been before my preparation routine looks more or less like this: ifconfig wlan0 list scan, copy SSID, vim /etc/wpa_supplicant.conf, service netif restart. The WM I use, i3, does not have fancy network managers that nicely offer you to join available WiFi networks. It’s mildly annoying but not annoying enough to actually make me open a browser and go looking for ways to automate this procedure. Implementing such a utility, on the other hand, sounds like a nice weekend project.

I wanted to try out the Go language for a long time but didn’t have a good idea for a project and this WiFi manager tool looked like a good fit to learn how stuff is done there because it required: starting a child process and parsing its output, file I/O, work with strings, TUI elements.

The data management bits were easy, the standard library provides all the required functionality, and for TUI framework I picked up termui. Midway through implementation, I realized that the main use case of the framework is to present data, and interacting with the user was not its forte, but I decided to press on and just hack something together. It was just for fun after all. It worked out acceptable in the end.

Without further ado, let me introduce Chai-Fi, the minimalistic wpa_supplicant.conf “manager” for FreeBSD. To be used in coffee shops, guest networks, and other locations. It’s got a bunch of stuff hardcoded, not very portable between different TERMs and probably won’t look nice with non-Unicode fonts. But hey, it works on my laptop. Who knows, might work on yours

If you want to jump right into the action check out freebsd-mkova repo. Below are some technical details on what OVA is and its internals.

In addition to installation media like DVD or memstick, FreeBSD RE ships FreeBSD releases as a virtual disk image in a number of formats. This is a convenient way when you start your whole VM using bhyve or qemu, as a single CLI command and pass the image location as an argument. It’s less handy if you try to create VM using GUI-based tools like VMWare or VirtualBox. You need to create VM and then configure it to use the image as a drive. Not an awful lot of work but still.

At my $DAYJOB I came across another way of shipping VM-based products called virtual appliance. Essentially it’s a pre-configured virtual machine with disk image included in a self-contained file that can be imported by hypervisor software using couple of mouse clicks or a single command like VBoxManage import. The file format for this kind of deliverables is OVA

Internally OVA is a tar(1) archive that includes OVF file, disk image or images in VMDK format, and optional manifest (list of files with checksums). OVF file is an XML file with VM description that looks like this. More information on the format can be found at

So on the surface, we need to take official VMDK, create XML file and package them both in .tar archive. Unfortunately, there is a problem: VMDK format comes in several flavors and while FreeBSD RE ships monolithic sparse, OVA needs the image in stream-optimized flavor.

As the name implies monolithic-sparse format does not have every single sector of the virtual disk in the file, it only stores the data that was written by OS. The standard unit of data on disk is a sector, usually 512 bytes long. VMDK organizes sectors in grains: a sequence of sectors. The size of the grain (in sectors) is defined in the VMDK header as a grainSize field. So if an image is N sectors long there are N/grainSize grains. Only grains with the content that is different from all-zeroes are stored in the image file. To organize fast random access to the stored data in the image VMDK maintains a two-level index: grain directory and grain table. Grain table is a fixed-size array (numGTEsPerGT field in the header) with offsets of the grain in VMDK. If offset is zero it means the grain is not present in the file and its content is all-zeroes. Grain tables’ offsets, in turn, are stored in grain directory. And again if offset in the directory is zero it means none of the grains in the table is present in the image. The sequence of steps to read sector X from the VMDK is approximately this:

grainIdx := [X / grainSize]                 // Find the grain index 
gtIdx := [grainIdx / numGTEsPerGt] // Find the grain table index
gtOffset := grainDirectory[gtIdx] // Find grain table offset // if gtOffset is zero return empty sector
grainTable := readAt(gtOffset) // read grain table data
grainOffset := grainTable[X % grainSize] // find grain offset // if grainOffset is zero return empty sector
sectorData := readAt(grainOffset + (X % grainSize)) // read sector data

The VMDK file layout for monolithic sparse format would look like:

[header] [imageDescriptor] [GrainDirectory] [GrainTable0] … [GrainTableN ] [ Grain0 ] [GrainX ] [ GrainY ] … [ GrainZ

The stream-optimized VMDK operates with the same concepts: grain, grain table, grain directory but organizes them differently. Again as the name implies it’s optimized for streaming which means no random access to the image file so it lays out elements a bit differently. GD, GT, grains are prefixed with the markers and can be detected as the software reads file from the network. Also, grains can be compressed to save bandwidth. The stream-optimized VMDK layout looks like this:

[ header ] [imageDescriptor] [GrainMarker] [Grain] … [GrainMarker] [Grain] [GrainTableMarker] [GrainTable0] … [GrainDirectoryMarker] [GrainDirectory] [FooterMarker][FinalHeader] [EndOfStreamMarker]

the FinalHeader is the copy of the header with the gdOffset field set to point to [GrainDirectory] location

As you can see the conversion from monolithic sparse to stream optimized is quite straightforward. I spent some time implementing it over the holiday break, added OVF generation logic and shipped freebsd-mkova. I wanted to write it in Go to practice a new language but was turned off by the XML generation and namespaces support in it, it felt unnecessary verbose and clunky. Will try to find some other project for Go my studies.

Audio for RK3399

November 30, 2019 — 3 Comments

Last two weeks I’ve been working on audio support for Firefly-RK3399. Full support requires a number of things that are not quite there or not available in the mainline FreeBSD kernel. The main low-level hardware functionality consists of two parts: I2S block in the SoC and RT5640 audiocodec that converts digital audio to an analog signal. They talk to each other using the I2S protocol. A little bit higher is FDT virtual “devices” called simple-audio-card. This part is responsible for coordinating the setup of both hardware components: make sure they agree on a number of channels, a number of bits per sample and clock specifics of the I2S protocol. There is no code for it in the FreeBSD kernel, so I had to just hardcode these things in both hardware drivers.

The other obviously missing part was I2S clocks in the CRU unit drivers. This was easily fixable by just consulting with RK3399 TRM. Still, having added the clocks support I couldn’t get the signal on the physical pin. Thanks to manu@ who pointed out that some stuff might be missing in io-domain and power regulators area it was resolved too after setting bit 1 of GRF_IO_VSEL register

With all these bits in place, I was able to get sound out of headphones, but it was distorted. My first instinct was to blame mismatch of clocks/formats between I2S and codec, so I spent two very frustrating days experimenting with setting polarities and data formats just to find out that there is GPIO that controls headphone output on Firefly-RK3399. This is Firefly-specific bit and only referenced in the Firefly fork of the Rockchip fork of the Linux kernel. The way it’s implemented loud noises still can pass through the filter to headphones, so at max volume, you can here laud parts as a series of pops and grunts. By manually configuring GPIO4_C5 (gpioc4/pin21) I finally was able to get clear sound out of the Firefly.

On the bright side now I know how to convert an I2S stream captured by a Saleae logic analyzer to wav file:

The next step is to check how much work would it take to implement the simple-audio-card part. The (very hacky) WIP is at

I was updating my laptop to the latest HEAD today and noticed that my bash prompt looks ugly in default console color scheme. So what with one thing and another I ended up writing color themes support for vt(4). Just because it was fun thing to do. The idea is that you can redefine any ANSI color in console using variable in /boot/loader.conf, i.e.:

kern.vt.color.0.rgb="0,0,0" # color 0 is black
# or
kern.vt.color.15.rgb="#ffffff" # color 15 is white

Here is how my Tomorrow Night theme looks like:


It works only with framebuffer-based console like efifb or i915kms.
Patch: console-color-theme.diff

Last year I got my hands on Minnowboard Turbot (courtesy of Frank H.) and spent some time working on communications protocols support for it. Below is short summary of what works and what doesn’t.

Minnowboard Turbot is Atom-base SoC, and standard x86 part (HDMI, network, USB) FreeBSD just works on it. The board has expansion connector that exposes I2C, SPI, and GPIO pins and can be used to talk to peripheral devices. Short summary of header pins can be found on


GPIO functionality on Minnowboard provided by bytgpio(4) driver (name was chosen to match OpenBSD one). There are three banks, but the most accessible is the third one available via /dev/gpioc2. All GPIO pins on the expansion header are pin number in this bank. So to set pin “GPIO 0” on the picture to HIGH you would use gpioc -f /dev/gpioc2 0 1.

To use some GPIO peripheral that has FreeBSD kernel driver you will have to describe it using hints in /boot/loader.conf. For instance there is LED D2 that is attached to pin 22. So to expose it through gpioled(4) interface you’d add following lines to /boot/loader.conf:

# pin 22: (1 << 22)


Driver for SPI support on Minnowboard is intelspi. Hardware supports all four SPI modes and clock frequency configuration but the driver doesn’t (yet). So it’s always mode 0 and frequency is hardcoded to 10MHz. To access it from userland you’d use spigen driver for now. As with gpio peripherlas, you’d have to add spigen via hints:


There is no spigen module, so it has to be enabled in kernel config:

device          spibus
device          spigen


I2C driver is ig4(4). There are two I2C buses, the one that is accessible from expansion header is iicbus0. As with previous two protocols devices should be defined by hints. For example:

# 0x68 in 8-bit format
# for example only, hasn't been tested

There is no ds3231 module so again, you need to enable it in kernel:

device          iicbus
device          ds3231

To use I2C from userland via /dev/iicN you’ll need to load iic(4) module.

Over years I accumulated fair number of devices I have no real use for. I ordered them either on impulse, or to add couple of $$$ to the bill to get free delivery. So in order to get at least some value from those purchases I put together goofy demo using two of such devices: I2C temperature sensor breakout from Sparkfun and 128×32 SPI OLED display from Adafruit.

I wrote two libraries to talk to TMP102 (chip in which temp sensor is based) over I2C and to SSD1306(OLED display chip) over SPI and several demos simple enough to put together in one day but flashy enough to excite my inner child and bring fond memories of MSDOS days. The SPI chip requires this fix in kernel. Userland SPI API provides only very basic functionality but luckily it was enough to talk to SSD1306.

TMP102 is I2C-only device but SSD1306 uses two signals in addition to standard SPI ones and GND/VCC: Data/Command switch and Reset. I connected them to GPIO pins 23 and 24 on my RPi. SSD1306 code is based on Adafruit’s Python library and supports only one model so far, but can easily be extended to support all three of them.


rfkill on Jetson TK1

December 11, 2016 — Leave a comment

If you’re trying to install half-sized mini-PCIe wifi card in Jetson TK1, be aware, that the board has rfkill feature that is enabled by default. rfkill is hardware or software controlled switch that enables/disables RF signal on the wifi card itself. In case of mini-PCIe it’s controlled by level on pin 20 of the card. On Jetson TK-1 that pin is connected to GPIO X.7 pin. So to enable wifi on the board you need to run something like:

gpioctl -cN gpio_X.7 OUT
gpioctl -N gpio_X.7 1

Update (11/29/2016): DTB overlay no longer required. PSCI monitor will patch in-memory DTB to add psci node. Commit

Short version of how to get RPi3 with SMP support

Long version of what’s in those .bin/.dtbo files

Boot sequence for PSCI monitor on RPi3 looks like this:

  • entry point is called at EL3
  • monitor performs some CPU-specific initializations
  • monitor sets up exceptions vector with SMC handler
  • monitor reserves memory so it’s not going to be overwritten by next boot stage
  • monitor drops to EL2, passes control to next boot stage

First problem arises at step #1. If you generate binary and boot it using kernel parameter in config.txt by the time control is passed to entry point CPU is already at EL2. It’s because VideoCore firmware does not pass control directly to kernel but runs some built-in code prior to that which is called ARM stub. Good news – that built-in code can be overridden. The history behind ARM stubs can be found in this thread. And source code for default ARMv8 stub is in armstub8.S file.
It can be overridden either by adding armstub=NNN.bin line to config.txt or by copying stub to armstub8.bin file on FAT partition, this file is loaded automatically if it’s present.

Default stub performs some initialization and runs following pseudocode on each CPU:

void *spin_entry[4];
if (current_cpu == 0)
else {
    while (spin_entry[current_cpu] == 0) {

It’s a good starting point so I added exception vector that jumps to PSCI handler routine, and PSCI entry points for PSCI_VERSION, CPU_ON, SYSTEM_REBOOT, SYSTEM_OFF, SYSTEM_RESET.

So far so good. Back to memory reservation. My initial idea was for PSCI to edit DTB blob. Either add one more memreserve block that starts at 0x0 and 16Kb long or edit /memory node and cut 16Kb from first reg pair. I ended up implementing both of these approaches and neither worked. Because on RPi3 boot sequence looks like: u-boot -> loader.efi -> kernel. All memory information is passed to kernel via EFI and this part of DTB is just ignored. And U-Boot does not care about FDT either. It gets available memory size by querying VideoCore directly using mailbox API and then initializes on DRAM bank: [0..dram_size].

After short while of heavy thinking I ended up with just passing amount of reserved memory as the second argument of U-Boots entry function and then using that amount to start DRAM bank range with. Hackish but works.

Now there is a matter of VideoCore-loaded FDT blob. Most of the FreeBSD images out there have device_tree_address=0x100 line in config.txt. This is reasonable value so FDT blob does not overlap with u-boot. Unfortunately new ARM stub with PSCI functionality is larger than 0x100 bytes and now overlaps with it. So device_tree_address has to be bumped to 0x4000. But U-Boot has 0x100 hardcoded as default so config.txt and u-boot now out of sync. On the other hand ARM stub (default and PSCI-enabled) does pass FDT blob address as the first argument to entry function. So u-boot can set value of fdt_addr_r to correct address dynamically.

PSCI: check, U-Boot: check. There is one more missing part. FreeBSD kernel identifies presence of PSCI by checking for FDT node with “arm,psci-0.2” compatibility string. Obviously there is no such thing in original dtb file. PSCI monitor adds this node dynamically by patching DTB blob in-memory before passing control to U-Boot. It would be natural to make PSCI monitor patch FDT blob and add this new node but I wasn’t up to the task of implementing it in assembler. I took the low road and just created DTB overlay with psci node. That’s why psci.dtbo and “dtoverlay=psci” line in config.txt are required. I’m planning to add C files to PSCI monitor and implement dynamic DTB patching but for now overlay should do.

Source code for all the stuff in this post: rpi3-psci-monitor and u-boot patches.

Precompiled bits: rpi3-smp