Archives For FreeBSD

Getting SMP support for RPI3 took some time and was interesting learning experience. So I share bits of what I learned here.

FreeBSD/arm boot start on one CPU (called primary) the rest of CPUs are “on hold”. At some point in boot sequence non-primary CPUs are forced to call mpentry() function. The way CPU is forced to call mpentry is platform-dependent. RPi2 for instance passes mpentry and argument to VideoCore using mailbox interface and then VideoCore kicks ARM CPU in action. So most of SoC has their own implementation of platform_mp_start_ap platform API method.

FreeBSD/arm64 takes more standardized approach. It relies on secure monitor to provide this functionality. To explain what secure monitor is I need to diverge a bit into Aarch64 architecture. Aarch64 has four Exception Levels: EL3, EL2, EL1, EL0. Higher level acts as a “kernel” to the lower level. Most usual roles for exception levels are:

EL3 – Secure Monitor
EL2 – Hypervisor
EL1 – OS kernel
EL0 – userland apps

Just like userland can request kernel service through syscal lower exception level can request service from higher one through syscall-like mechanism. On Aarch64 it’s SMC, HVC, SVC instruction to get to EL3, EL2, EL1 respectively. There are multiple services that EL1 can request from Hypervisor or Secure Monitor but one of them is PSCI (Power State Coordination Interface). It’s an API to control CPUs and power states of the system. Full spec can be found here.

SMP support on FreeBSD/arm64 requires Secure Monitor and there is no such thing for Raspberry Pi 3. Normally it’s a piece of software that is provided as a part of SoC. Stage 1 boot loader resides in secure ROM and every subsequent boot stage can be verified using certificates. There are regions of memory where secure monitor is loaded that can not be accessed from non-secure exception levels. No such thing on RPi3 either.

There were 3 options to get PSCI on RPi3: write “bare metal” implementation from the scratch, add it to U-Boot or port existing open source secure monitor to RPi3. Since U-Boot was already a part of the setup I looked up if there was PSCI support for it. There was for ARMv7 in the mainline and for AMRv8 as a patchset with generic PSCI framework and implementation for one of the platforms. Framework relied on the existence of secure memory region which is not a thing on RPi3 so I went for another option: port existing monitor.

Reference implementation of SM is ARM Trusted Firmware. It’s very well documented and easily extendable but the problem was that just like with U-Boot it made assumptions about memory layout that were not true for RPi3 and were not easy to change in the ATF code.

At this point bare metal PSCI emulation looked like less work than the other two options and I started experimenting with really low level stuff. I will write my progress up in the next blog post.

MFC tracking web-tool

October 16, 2016 — Leave a comment

Next week I plan to do bunch of MFCs for ARM and evdev stuff and to make it less of an ordeal I made this tool to track what’s merged and what’s not: mfc.kernelnomicon.org. It provides basic functionality I thought I would need for this process:

  • Navigator for HEAD commits with visual representation of MFC state: “no mfc scheduled”, “ready for mfc”, “waiting for ‘mfc after’ date”, “merged”
  • Basic filtering: by author name, by two states: “waiting”, and “ready”
  • “MFC basket” – manage set of commits I would like to merge back in one go
  • Generate svn command and change log for selected “MFC basket”

Stack used: Django + Bootstrap3 + jQuery

GitHub project: mfctracker

Raspberry Pi 3 limited support was committed to HEAD. Most of drivers should work with upstream dtb, RNG requires attention because callout mode seems to be broken and there is no IRQ in upstream device tree file. SMP is work in progress. There are some compatibility issue with VCHIQ driver due to some assumptions that are true only for ARM platform.

SD card layout is the same as for RPi and RPi2 but boot chain is different. All ARM64 supported by FreeBSD up to now used EFI as boot environment. RPi 3 has only VC firmware and whatever it can spin off, e.g. u-boot. So it seemed easier to enable EFI API in U-Boot instead of porting ubldr to arm64. There were some hiccups with netbooting, (see patch) but otherwise it was OK. U-Boot port and crochet config for Pi 3 should be committed “real soon”(tm).

For those who would like to try it ASAP Shawn Webb put together instruction on how to get bootable SD image.

Short summary of couple of frustrating hours I spent trying to get my RPi3 netbooting:

U-Boot – 2016.09
raspberrypi/firmware/boot – ec63df146f454e8cab7080380f9138246d877013

armstub.bin, armstub7.bin, armstub8.bin – NOT REQUIRED. There are tens of google results mentioning these files along with some dd magic – they all obsolete. Aforementioned version of firmware does not require them. 64-bit mode is controlled by arm_control variable in config.txt (see below). If 64-bit mode requested default kernel name becomes kernel8.img, and kernel load address becomes 0x80000. U-Boot uses correct default load address so no need for any additional parameters or hacks.

Building u-boot:

# gmake ARCH=arm CROSS_COMPILE=aarch64-none-elf- CONFIG_EFI=y rpi_3_defconfig
# gmake ARCH=arm CROSS_COMPILE=aarch64-none-elf- CONFIG_EFI=y 

Boot partition:
– Copy all files/dirs from firmware/boot to your FAT partition (let’s say it’s mounted at /mnt/fat)
– Copy u-boot.bin to kernel8.img on your FAT partition
– Create config.txt with following content:

arm_control=0x200
enable_uart=1

If you want to keep u-boot.bin name and not use kernel8.img, add following ling:

kernel=u-boot.bin

Unmount and boot RPi3. You should see U-Boot output on both serial console and HDMI.

Default U-Boot build uses mini-uart. So if for some reason you want to use PL011 instead, patch include/configs/rpi.h and make sure that CONFIG_BCM283X_MU_SERIAL is not defined and CONFIG_PL01X_SERIAL is defined instead.

config.txt in this case should look like:

arm_control=0x200
dtoverlay=pi3-disable-bt

Few weeks ago evdev support was finally committed to HEAD. Project started a part of SoC 2014 by Jakub Klama and then picked up, finished and submitted by Vladimir Kondratiev. It’s drop-in compatible with Linux API which means all you need to do is add #ifdef __FreeBSD_ around respective includes and existing code (if it’s otherwise cross-compatible with FreeBSD) should just work. Which is the case for Qt and to lesser extent for tslib. Hardware support is still moving target, FreeBSD has evdev-compatible drivers for USB keyboards, USB mice, TI’s AM33xx touchscreen controller and Raspberry Pi’s official touchscreen. Only the latter device supports multitouch and Vladimir submitted patch required to get it working. To my knowledge it’s the first multitouch touchscreen ever working on FreeBSD so I decided to record demo to save this moment for generations to come. Well, not really. Mostly to brag and to let people know that it’s possible and encourage them to make stuff and experiment with FreeBSD, ARM, and Qt.

Demo below is standard imagegestures example built using latest dev branch of Qt.

I spent Labor Day weekend laboring on VBox shared folders support for FreeBSD. It’s been some time since I worked on it last time so I had to refresh my memory first. Things have moved on since then – VBox in ports was updated to version 5, but fortunately Li-Wen synced up freebsd-vboxfs repo to the latest version. After three days of laid-back hacking I am glad to announce that following VOPs are kind of implemented (in no particular order): lookup, access, readdir, read, getattr, readlink, remove, rmdir, symlink, close, create, open, write. “Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host’s md5sum, create/remove directories, unzip zip file, etc but I doubt it would survive stress-test. Locking is all wrong at the moment and read/write VOPs allocate buffers for every operation.

I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them in vboxfs nodes, but if one of parent directories is renamed, all cached names should be modified accordingly. I am going to tackle these two problems once I have long enough stretch of time time sit and concentrate on task.

Looks like my attempt to cheap out on SSD for TK1 has backfired. I went for the cheapest SSD available in local store (Toshiba Q300) but when I tried to checkout FreeBSD sources to the drive I got bunch of WRITE_FPDMA_QUEUED timeouts and system locked up. The same thing happened when I tried to perform checkout on Linux. The drive itself was OK, it survived “svn co …/head” and dd when connected using USB-to-SATA adapter.

I believe the problem was that TK1 SATA voltage was out of Toshiba’s tolerance range. I replaced Q300 with Samsung EVO 850 and was able to checkout sources and finish buildworld using SSD for src/obj storage.

FreeBSD on Jetson TK1

June 28, 2016 — 4 Comments

I finally got around to BSDify my Jetson TK1. Here is short summary of what is involved. And to save you some scrolling here are artifacts obtained from whole ordeal: https://people.freebsd.org/~gonzo/arm/jetson-tk1/

Jetson TK1

U-Boot

First of all – my TK1 didn’t have U-Boot. Type of bootloader depends on the version of Linux4Tegra TK1 comes with. Mine had L4T R19, with some kind of “not u-boot” bootloader. My first attempt was to use tegrarcm tool, it uses libusb, so it’s possible to build it on FreeBSD with some elbow grease, but once I tried to run it – it gave me cryptic errors and USB is not my strong skill so I took low road and installed Ubuntu VM. For what is’s worth I got the same kind of error on Ubuntu.

Next step was to use official update procedure described in http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/l4t_quick_start_guide.txt. Since I wasn’t going to boot Linux on the board I didn’t need sample rootfs. So the whole procedure was:

– Go to L4T R21.4 page
– Download Tegra124_Linux_R21.4.0_armhf.tbz2
– Unpack it
– Connect microUSB port on device to Linux VM
– Get device into recover mode: power cycle, press and hold recovery button, press and release power button, release recovery button
– Run ./flash.sh jetson-tk1 mmcblk0p1, this should rewrite eMMC flash on the board and after reboot you will get u-boot prompt on serial console

FreeBSD

At this point you can boot FreeBSD on TK1. I use netboot for most of my device so in this case it was: build and deploy world to /src/FreeBSD/tftproot/tk1, build and install kernel to the same directory, copy /src/FreeBSD/tftproot/tk1/boot/kernel/kernel to kernel.TK1 in tftproot directory, add entry do DHCP config and restart DHCP server. Entry looks like this:

host tk1 {
        hardware ethernet 00:04:4b:49:08:9e;
        fixed-address 192.168.10.98;
        filename "kernel.TK1";
        option root-path "/src/FreeBSD/tftproot/tk1";
        option root-opts "nolockd";
        option routers 192.168.10.1;
}

And also you need to add this to sys/arm/conf/JETSON-TK1 before building kernel:

options        BOOTP
options        BOOTP_NFSROOT
options        BOOTP_COMPAT
options        BOOTP_NFSV3

On the device you just run “dhcp; bootelf” and voila – it just works.

ubldr

Next step was to get ubldr running. I prefer suing ubldr because it gives more control over boot process accessible from booted FreeBSD system. ubldr requires U-Boot with API support, so I had to rebuild U-Boot from sources provided by nvidia with added #define CONFIG_API and all standard patches from sysutils/u-boot-* ports. Build procedure is standard:

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make jetson-tk1_config
make

It will generate multiple files, u-boot-dtb-tegra.bin is the one you want.

To reflash board with non-standard u-boot run ./flash.sh -L /path/to/u-boot-dtb-tegra.bin jetson-tk1 mmcblk0p1

Back to ubldr. It was easy to build and load it. Build script:

#!/bin/sh
export TARGET=arm
export TARGET_ARCH=armv6
export SRCROOT=/src/FreeBSD/wip
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj
export MAKESYSPATH=$SRCROOT/share/mk

set -x
set -e

buildenv=`make -C $SRCROOT TARGET_ARCH=armv6 buildenvvars`
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH obj
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH clean
eval $buildenv make -C $SRCROOT/sys/boot -m $MAKESYSPATH UBLDR_LOADADDR=0x80600000 all

sudo cp /src/FreeBSD/obj/arm.armv6/src/FreeBSD/wip/sys/boot/arm/uboot/ubldr /src/FreeBSD/tftpboot/ubldr.TK1

Obviously, kernel.TK1 in DHCP config needs to be replaced with ubldr.TK1. 0x80600000 is some value I came up with by looking at u-boot default environment. Something not high enough to overlap with kernel and not low enough to overlap with u-boot.

And that’s where thing got hairy. To load ubldr and then netboot kernel, you need to set u-boot env loaderdev variable first: setenv loaderdev net; saveenv. And then do the same thing as above: dhcp; bootelf. Unfortunately I got this:

## Starting application at 0x81000098 ...
Consoles: U-Boot console
Compatible U-Boot API signature found @0xffa3e410

FreeBSD/armv6 U-Boot loader, Revision 1.2
(gonzo@eb3.bluezbox.com, Mon Jun 27 19:59:22 PDT 2016)

DRAM: 2048MB
MMC: no card present
MMC Device 2 not found
MMC Device 3 not found
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC: no card present
MMC Device 2 not found
Number of U-Boot devices: 3
U-Boot env: loaderdev='net'
Found U-Boot device: disk
Found U-Boot device: net
Booting from net0:
panic: arp: no response for 192.168.10.1

--> Press a key on the console to reboot <--
Rebooting...
resetting ...

After some heavy thinking and code digging problem was narrowed down to u-boot network driver drivers/net/rtl8169.c. Instead of returning 0 on success and negative value on error it returns number of bytes sent on success and zero on error. Which confused ubldr into thinking nothing is sent, so recv part of exchange was never invoked. After fixing this issue kernel was loaded just fine but hang right afert

Using DTB compiled into kernel.
Kernel entry at 0x0x80800100...
Kernel args: (null)

Logn story short - it was caused by enabled D-Cache so I had to add

#ifndef CONFIG_SPL_BUILD
#define CONFIG_SYS_DCACHE_OFF
#define CONFIG_CMD_CACHE
#endif

to u-boot config and go through rebuild/reflash cycle again. After this whole boot chain went through right to login prompt.

My next goal is to make TK1 self-contained box: get base system installed on eMMC and use attached SSD as scratch disk for swap and builds.

Few weeks back Ralf Nolden, who is *BSD champion in Qt community, urged me to clean-up and submit my Qt5-related projects to upstream and scfb platform plugin was picked as a test dummy. It took 12 iterations to get things right, along the way plugin was renamed to bsdfb, but eventually patch has been merged.

Next two candidates are bsdkeyboard and bsdsysmouse input plugins.

Two months ago I tried to setup dev environment using FreeBSD Vagrant box just to find out that FreeBSD does not support VirtualBox shared folders. After some googling I found Li-Wen Hsu’s github repository with some work in this area. Li-Wen and Will Andrews has already done major chunk of work: patches to VirtualBox build system, skeleton VFS driver, API to talk to hypervisor but hit a block with some implementation details in VirtualBox’s virtual-memory compatibility layer. Will provided very comprehensive analysis of the problem.

Li-Wen was occupied with some other projects so he gave me his OK to work on shared folder support on my own. Will’s suggestion was easy to implement – lock only userland memory, like Solaris driver does. VFS part was more complicated though: fs nodes, vnode, their lifecycle and locking is too hairy for drive-by hacking. I used tmpfs as a reference to learn some VFS magic, but a lot of things are still obscure. Nevertheless after few weeks of tinkering first milestone has been achieved: I can mount/unmount shared VirtualBox folder and navigate mounted filesystem without immediate kernel panic. Next goal (if time permits): stable and non-leaking read-only filesystem.