Archives For Uncategorized

A week ago Adrian Chadd asked me to take a loot at FreeBSD/MIPS emulation. So last week I’ve been busy tidying up stuff in that department and looking up bits of information on various emulators. This morning I finally committed last changeset so now is the time to write up summary.

Emulators

There are two widely used MIPS emulatoes that FreeBSD supports: QEMU and GXemul. Both of them support numerous MIPS devices but we’re interested in only two.

MALTA is more or less standard for MIPS emulation and supported by both emulators. QEMU supports 32 and 64 bit variants with both big and little-endian byte order. So four modes in total. Also for MALTA machine QEMU provides PCNet NIC emulation.

GXemul supports 32 and 64 bit modes of MALTA but only for little-endian byte order. Big-endian byte order is not supported due to incomplete PCI controller implementation. No NIC support for MALTA machine. Also Gxemul provides so-called oldtestmips emulation mode: generic implementation of abstract MIPS machine with very simplified NIC/disk devices. In theory it should be faster then actual hardware emulation but I haven’t got around to benchmarking yet. oldtestmips can be run in 32 and 64 bit mode, but only big-endian byte order is supported.

Disk images

I used raw disk images created by makefs(8) utility. The advantage of raw disk image is that they can be used with both emulators. qemu provides more options in this area but they’re out of scope of this article. I created four images for my tests: disk.img, disk64.img, diskel.img, and disk64el.img. 32-bit big-endian, 64-bit big-endian, 32-bit little-endian, 64-bit big-endian. The process looks somewhat like this script:

#!/bin/sh
set -e

export TARGET=mips
export TARGET_ARCH=mips
export SRCCONF=/dev/null
export SRCROOT=/src/FreeBSD/head
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj/head
export DESTDIR=/src/FreeBSD/tftproot/$TARGET_ARCH
make -C $SRCROOT buildworld

sudo -E mkdir -p /src/FreeBSD/tftproot/$TARGET_ARCH
sudo -E make -C $SRCROOT KERNCONF=$KERNCONF DESTDIR=$DESTDIR installworld
sudo -E make -C $SRCROOT KERNCONF=$KERNCONF DESTDIR=$DESTDIR distribution

# modify /etc/fstab and /etc/rc.conf in $DESTDIR

# Create 512Mb disk image with big-endian UFS
# For TARGET_ARCH set to mipsel or mips64el use "-B le" switch 
sudo -E makefs -M 538968064 -B be /src/FreeBSD/disk.img $DESTDIR

Change TARGET_ARCH and disk image name accordingly for mipsel/mip64/mips64el targets.

QEMU

As of QEMU 1.6.0 there is one known problem with it: NIC emulation does not work in big-endian mode. See this patch for fix and details. Also, amount of memory limited to 128Mb. Use MALTA kernel config for 32-bit mode and MALTA64 for 64-bit mode.

Command lines for various modes/byte orders are:

qemu-system-mips -M malta -kernel /path/to/mips.mips/MALTA/.../kernel -hda /src/FreeBSD/disk.img -nographic
qemu-system-mips64 -M malta -kernel /path/to/mips.mips64/.../MALTA64/kernel -hda /src/FreeBSD/disk64.img -nographic
qemu-system-mipsel -M malta -kernel /path/to/mips.mipsel/.../MALTA/kernel -hda /src/FreeBSD/diskel.img -nographic
qemu-system-mips64el -M malta -kernel /path/to/mips.mips64el/.../MALTA64/kernel -hda /src/FreeBSD/disk64el.img -nographic

GXemul

GXemul requires two patches for proper FreeBSD/MIPS emulation. First one implements rdhwr op for reg 29 required by TLS support. The second one fixes UDP packet checksum calculation and required for proper functioning of emulated DHCP server. Both of these patches have been committed upstream but there were no GXemul release after that so they’re not available as part of emulators/gxemul port.

As with QEMU kernel configs for MALTA are MALTA and MALTA64. And since only little-endian byte order is supported command lines for this emulation mode are:

gxemul -e malta -C 4Kc -d /home/gonzo/FreeBSD/diskel.img /path/to/mips.mipsel/.../MALTA/kernel
gxemul -e malta -d /home/gonzo/FreeBSD/disk64el.img /path/to/mips.mips64el/.../MALTA64/kernel

For oldtestmips emulation mode is other way around, only big-endian. And to make things more complicated it’s GXEMUL kernel config for 64-bit and GXEMUL32 for 32-bit. So here you are:

gxemul -M 256 -E oldtestmips -d /home/gonzo/FreeBSD/disk64.img /path/to/mips.mips64/.../GXEMUL/kernel
gxemul -M 256 -C 4Kc -E oldtestmips -d /home/gonzo/FreeBSD/disk.img /path/to/mips.mips/.../GXEMUL32/kernel

Maximum amount of memory set could be set by -M command-line switch for oldtestmips emulation is 256M. Otherwise physical memory would overlap with memory-mapped regions reserved by devices. There are some preliminary work to work around this limitation but no results yet.

It took me two months but I finally got back to hacking on musb driver for FreeBSD (the one that is used in TI AM335x-based devices like Beaglebone or Beaglebone Black). Previous revision turned out not to be ready for production. Here is the new one: beaglebone-usb-20130626.diff. I adopted it to latest HEAD, fixed numerous bugs, added support for SPLIT transactions and USB suspend/resume signalling. There is some cleaning-up to do but unless something major comes up the plan is to commit it over next few days.

I finally got around to finishing the PWM and LCDC driver for AM335x. Everything was committed today. Here is demo on AM335x EVM (I apologize for quality):

IMG_0049

As it was mentioned in previous post U-Boot can boot FreeBSD kernel directly but this approach doesn’t allow a great deal of control over boot process: there is no way to set tunables’ values or pre-load module. Controlling this stuff requires more knowledge of FreeBSD internal data structures and its boot process then U-Boot holds.

On i386 and other Tier1 architectures this task is handled by the loader(8) program. It’s last stage boot loader (e.g. it’s supposed to pass control to FreeBSD kernel only), highly customizable and scriptable. loader(8) relies on one of the previous stages boot loader to access resources like disks, console, network. For i386 it’s BTX and BIOS.

ubldr is implementation of loader(8) for ARM on top of U-Boot. Original code was developed by Semihalf back in 2008 and has been being improved by them and FreeBSD Community since then. Despite it has been around for almost 5 years amount of documentation is shockingly low. I found only these slides from BSDCan 2008.

One of the nice feature U-Boot provides is API for stand-alone process. If you don’t need full-blown operating system running on your hardware but still want access to SD card/network/console you can request them from U-Boot via syscall-like API that turns boot loader into quasi-OS. Some bits of information on this topic can be found in api/README file in U-Boot sources: here.

ubldr uses U-Boot API to enumerate devices that might be used as a boot source: block (e.g. SD card) or network. For network device it will use BOOTP to try to obtain network/boot data and then mount directory over NFS. For block device it will inspect partition table and try to find suitable partition to use as a root device. Once root is mounted ubldr will perform standard loader(8) magic: get loader-related config from /boot/ directory and act on it.

There is no dedicated top-level build target for ubldr so getting it compiled is a little bit tricky. You need to perform whole buildworld cycle before compiling ubldr. Build script would look something like this:

export SRCROOT=/src/FreeBSD/head
export MAKESYSPATH=$SRCROOT/share/mk
export TARGET=arm
export TARGET_ARCH=armv6
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj

make -C $SRCROOT buildworld

buildenv=`make -C $SRCROOT buildenvvars`

eval $buildenv make -C $SRCROOT/sys/boot clean
eval $buildenv make -C $SRCROOT/sys/boot obj
eval $buildenv make -C $SRCROOT/sys/boot UBLDR_LOADADDR=0x2000000 all

Meaning of UBLDR_LOADADDR is the same as KERNPHYSADDR in previous post.

ubldr is ELF executable and can be used with bootelf command. Typical boot log is something like this:

## Starting application at 0x02000054 ...
Consoles: U-Boot console  
Compatible API signature found @7b662a8
Number of U-Boot devices: 2

FreeBSD/armv6 U-Boot loader, Revision 1.2
(gonzo@bsdbox, Fri Apr 19 18:52:33 PDT 2013)
DRAM:    128MB

Device: disk

Device: net

/boot/kernel/kernel data=0x3ae624+0x2128c syms=[0x4+0x71ca0+0x4+0x44075]
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Waiting for Ethernet connection... done.
Using DTB provided by U-Boot.
Kernel entry at 0x100100...
Kernel args: (null)
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
...

I was asked to share details about my root-over-NFS setup so here they are. I decided to split how-to in two posts: server/kernel part and u-boot part.

Usual components in the setup are:

  • DHCP server
  • TFTP server
  • NFS server
  • NAT (optional)

DHCP server

I use net/isc-dhcp42-server as a server. Sample dhcpd.conf:

option root-opts code 130 = string; # NFS / mount options
log-facility local7;

subnet 192.168.10.0 netmask 255.255.255.0 {
        server-name "cinderella.bluezbox.com";
        server-identifier 192.168.10.1;
        option subnet-mask 255.255.255.0;
        option broadcast-address 192.168.10.255;
        option domain-name-servers 8.8.8.8;
        option domain-name "bluezbox.com";
        next-server 192.168.10.1;
        option routers 192.168.10.1;
}

group {
        host pandaboard {
                hardware ethernet 0E:60:33:B1:46:01;
                fixed-address 192.168.10.90;
                filename "kernel.PANDA.bin";
                option root-path "/src/FreeBSD/nfs/armv6";
                option root-opts "nolockd";
        }

        host rpi {
                hardware ethernet b8:27:eb:f6:08:83;
                fixed-address 192.168.10.91;
                filename "ubldr";
                option root-path "/src/FreeBSD/nfs/rpi";
                option root-opts "nolockd";
        }
}

Config is pretty self-explanatory. I use google’s 8.8.8.8 nameserver but you can change it to your very own DNS server. Difference between various filename “…” will be explained later.

dhcpd should be enabled in rc.conf(5)

dhcpd_enable="YES"

TFTP server

TFTP server provides access to all files described in filename “…” options so it’s better to keep them together. By default it’s /tftpboot directory but I have whole drive dedicated to FreeBSD development environment and mounted under /src/FreeBSD mountpoint. So I keep everything there and my TFTP server root is /src/FreeBSD/tftpboot. TFTP server is standard FreeBSD’s one and config line in inetd.conf(8) looks like:

tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /src/FreeBSD/tftpboot

inted should be enabled in rc.conf(5)

inetd_enable="YES"

NFS Server

NFS server is not just one daemon but several services combined. So rc.conf(5) part of config looks like this:

rpcbind_enable="YES"
rpc_statd_enable="YES"
rpc_lockd_enable="YES"
nfs_server_enable="YES"
mountd_enable="YES"

Filesystems that are exported via NFS listed in exports(5). Mine contains following:

# Mind mount points obj and nfs are different
/usr/ports -maproot=0 -network 192.168.10.0/16
/src/FreeBSD/head /src/FreeBSD/nfs/rpi /src/FreeBSD/nfs/armv6 /src/FreeBSD/nfs/am335x -maproot=0 -network 192.168.10.0/16

Note that you can join several directories into one line only if they belong to the same mount point.

NAT

If you’re planning on building ports on the device – you’ll need internet access on it. All my devices are restricted to one LAN with laptop acting as a gateway. I use pf(4) for NATing. Config:
/etc/pf.conf(5)

ext_if=em0
rede="{192.168.0.0/16}"

nat on $ext_if from $rede to any -> ($ext_if)

and rc.conf(5)

gateway_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""

Kernel config

FreeBSD kernel should be properly configured in order to be suitable for mounting root over NFS:

options         NFSCL
options         NFSCLIENT # NFS v3
options         NFS_ROOT
options         BOOTP_NFSROOT
options         BOOTP_COMPAT
options         BOOTP
options         BOOTP_NFSV3
options         BOOTP_WIRED_TO=ue0

BOOTP_WIRED_TO value is SoC-specific. If you do not have full control over your DHCP server (e.g. it’s cable modem) and can’t specify root-path/root-opts you still can hardcode root location by removing BOOTP_NFSROOT and adding

options         ROOTDEVNAME=\"nfs:192.168.10:/src/FreeBSD/nfs/rpi\"

Installation

Depending on your boot sequence installation consists of one or two steps.

Normal system installation, e.g.:

sudo -E make TARGET_ARCH=armv6 DESTDIR=/src/FreeBSD/nfs/rpi -DDB_FROM_SRC installworld
sudo -E make TARGET_ARCH=armv6 DESTDIR=/src/FreeBSD/nfs/rpi -DDB_FROM_SRC distribution

And installing kernel copying kernel to tftpboot directory:

sudo -E make TARGET_ARCH=armv6 DESTDIR=/src/FreeBSD/nfs/rpi -DDB_FROM_SRC installkernel
cp /src/FreeBSD/nfs/rpi/boot/kernel/kernel /src/FreeBSD/tftpboot/kernel.RPI 

In some cases you’d want to use kernel.bin instead of kernel (more on it in next post) so second step would look like

sudo -E make TARGET_ARCH=armv6 DESTDIR=/src/FreeBSD/nfs/rpi -DDB_FROM_SRC KERNEL_EXTRA_INSTALL=kernel.bin installkernel
cp /src/FreeBSD/nfs/rpi/boot/kernel/kernel.bin /src/FreeBSD/tftpboot/kernel.RPI.bin

And if you’re going to use ubldr, there is no need to copy installed kernel anywhere.

More details on different types of binaries and boot process in the next post

Last few weeks I’ve been acting as a reviewer for Ganbold Tsagaankhuu’s port of FreeBSD for Cubieboard so in order to provide more valuable input and less naysaying I decided to get A10-based device to test his changes. So I ordered Hackberry from miniand.com. I’m not great fan of pushing SD cards back and forth so first thing I do with my SoCs is get them netbooting. That’s where fun begins.

Long story short – I had to get latest official u-boot, merge network driver (sunxi_wemac) from hno’s u-boot and add some GPIO magic to emac initialization. Namely – configure pin H19 (emac_power) as function 1 and set its value to 1. Only then I got net-related commands working properly.

If you’re interested in building your own bootable SD card: fetch these files. uEnv.txt and boot.scr are tailored to my needs so you might want to change them. mksd.sh does all the job, just make sure you use proper device name for SD card.

I started this smallish project to overcome coder’s block and original idea was to implement it in only one scripting language. I chose Python as I believed it had most clear API but then I decided to throw in Perl and Ruby as well. It was more exercise in building C extensions then in actual system programming but it was fun nonetheless. All three sub-projects lack proper documentation but there are examples that should be enough to get started.

Sources available on github.

Update 1: Add -DDB_FROM_SRC to install targets
Update 2: Add user “pi” with password “raspberry”
Update 3: Added host system requirements suggested in comments
Update 4: Rename bcm2835-rpi-b.dtb to rpi.dtb

It’s been a while since I posted original build instruction and a lot has change. Here is new version of it with some explanations:

First make sure your host system is configured properly:

  • WITHOUT_FORTH= must NOT be set in src.conf
  • msdos support must be available in the kernel
  • geom_md support must be available in the kernel

All code has been moved to HEAD. freebsd-pi repository on github serves only historical purpose and I guess will be removed at some point in future. So in order to build image you need sources for -head

svn co svn://svn.freebsd.org/base/head

TARGET_CPUTYPE is dropped to in favor of TARGET_ARCH=armv6. U-Boot, firmware files and boot process were upgraded too. Current boot chain is as follows:

  • Pi is powered up and loads firmware files
  • Firmware loads FDT blob defined in config.txt as device_tree variable at address defined as device_tree_address and fixes up fields like memory size, clock frequency and MAC address
  • Firmware loads u-boot and passes control to it
  • U-boot loads boot.scr and executes it
  • Default boot.scr loads ubldr(loader(8)-compatible implementation built over U-Boot API) and passes control
  • ubldr checks FreeBSD partition for /boot/loader.rc, loads it, the loads /boot/kernel/kernel and passes control to it
  • loader.rc should contain “fdt addr 0×100″ command. It will pass FDT blob filled in step #2 to kernel

So updated build script will look something like this:

#!/bin/sh
set -e

# Change this
export GPU_MEM=128
export PI_USER=pi
export PI_USER_PASSWORD=raspberry
export SRCROOT=/src/FreeBSD/head
export MNTDIR=/mnt
export MAKEOBJDIRPREFIX=/src/FreeBSD/obj
export IMG=/src/FreeBSD/obj/bsd-pi.img

export TARGET_ARCH=armv6
export MAKESYSPATH=$SRCROOT/share/mk
export KERNCONF=RPI-B

if [ -z "$MNTDIR" ]; then
        echo "MNTDIR is not set properly"
        exit 1
fi

KERNEL=`realpath $MAKEOBJDIRPREFIX`/arm.armv6/`realpath $SRCROOT`/sys/$KERNCONF/kernel
UBLDR=`realpath $MAKEOBJDIRPREFIX`/arm.armv6/`realpath $SRCROOT`/sys/boot/arm/uboot/ubldr
DTB=`realpath $MAKEOBJDIRPREFIX`/arm.armv6/`realpath $SRCROOT`/sys/$KERNCONF/rpi.dtb

make -C $SRCROOT kernel-toolchain
make -C $SRCROOT KERNCONF=$KERNCONF WITH_FDT=yes buildkernel
make -C $SRCROOT MALLOC_PRODUCTION=yes buildworld

buildenv=`make -C $SRCROOT buildenvvars`

eval $buildenv make -C $SRCROOT/sys/boot clean
eval $buildenv make -C $SRCROOT/sys/boot obj
eval $buildenv make -C $SRCROOT/sys/boot UBLDR_LOADADDR=0x2000000 all

rm -f $IMG
dd if=/dev/zero of=$IMG bs=128M count=8
MDFILE=`mdconfig -a -f $IMG`
gpart create -s MBR ${MDFILE}

# Boot partition
gpart add -s 32m -t '!12' ${MDFILE}
gpart set -a active -i 1 ${MDFILE}
newfs_msdos -L boot -F 16 /dev/${MDFILE}s1
mount_msdosfs /dev/${MDFILE}s1 $MNTDIR
fetch -q -o - http://people.freebsd.org/~gonzo/arm/rpi/freebsd-uboot-20130201.tar.gz | tar -x -v -z -C $MNTDIR -f -

cat >> $MNTDIR/config.txt <<__EOC__
gpu_mem=$GPU_MEM
device_tree=devtree.dat
device_tree_address=0x100
disable_commandline_tags=1
__EOC__
cp $UBLDR $MNTDIR
cp $DTB $MNTDIR/devtree.dat
umount $MNTDIR

# FreeBSD partition
gpart add -t freebsd ${MDFILE}
gpart create -s BSD ${MDFILE}s2
gpart add -t freebsd-ufs ${MDFILE}s2
newfs /dev/${MDFILE}s2a

# Turn on Softupdates
tunefs -n enable /dev/${MDFILE}s2a
# Turn on SUJ with a minimally-sized journal.
# This makes reboots tolerable if you just pull power on the BB
# Note:  A slow SDHC reads about 1MB/s, so a 30MB
# journal can delay boot by 30s.
tunefs -j enable -S 4194304 /dev/${MDFILE}s2a
# Turn on NFSv4 ACLs
tunefs -N enable /dev/${MDFILE}s2a

mount /dev/${MDFILE}s2a $MNTDIR

make -C $SRCROOT DESTDIR=$MNTDIR -DDB_FROM_SRC installkernel
make -C $SRCROOT DESTDIR=$MNTDIR -DDB_FROM_SRC installworld
make -C $SRCROOT DESTDIR=$MNTDIR -DDB_FROM_SRC distribution

echo 'fdt addr 0x100' > $MNTDIR/boot/loader.rc

echo '/dev/mmcsd0s2a / ufs rw,noatime 1 1' > $MNTDIR/etc/fstab

cat > $MNTDIR/etc/rc.conf <<__EORC__
hostname="raspberry-pi"
ifconfig_ue0="DHCP"
sshd_enable="YES"

devd_enable="YES"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
__EORC__

cat > $MNTDIR/etc/ttys <<__EOTTYS__
ttyv0 "/usr/libexec/getty Pc" xterm on secure
ttyv1 "/usr/libexec/getty Pc" xterm on secure
ttyv2 "/usr/libexec/getty Pc" xterm on secure
ttyv3 "/usr/libexec/getty Pc" xterm on secure
ttyv4 "/usr/libexec/getty Pc" xterm on secure
ttyv5 "/usr/libexec/getty Pc" xterm on secure
ttyv6 "/usr/libexec/getty Pc" xterm on secure
ttyu0 "/usr/libexec/getty 3wire.115200" dialup on secure
__EOTTYS__

echo $PI_USER_PASSWORD | pw -V $MNTDIR/etc useradd -h 0 -n $PI_USER -c "Raspberry Pi User" -s /bin/csh -m
pw -V $MNTDIR/etc groupmod wheel -m $PI_USER
PI_USER_UID=`pw -V $MNTDIR/etc usershow $PI_USER | cut -f 3 -d :`
PI_USER_GID=`pw -V $MNTDIR/etc usershow $PI_USER | cut -f 4 -d :`
mkdir -p $MNTDIR/home/$PI_USER
chown $PI_USER_UID:$PI_USER_GID $MNTDIR/home/$PI_USER

umount $MNTDIR
mdconfig -d -u $MDFILE


I finally got around to setting up experimental pkgng repo for ARM in order to share packages with other ARM developers and users who feel adventurous. And man, was it simple. I have pandaboard that is super-fast comparing to Raspberri Pi so I use it for building ports. There were several installed so I just had to generate packages for them using

pkg create -a

command. Then I uploaded all newly generated files to the server, grabbed packages built and shared by Stephen Hurd, removed duplicates with older versions and generated repo.txz by issuing

pkg repo

command.

Then on a raspberry pi I created pkg.conf in which I pointed to my newly created “repo”, updated metada and installed git:

# echo 'PACKAGESITE: http://people.freebsd.org/~gonzo/arm/pkg/' > /usr/local/etc/pkg.conf
# pkg update
# pkg install git

Pi took some time to push files back and forth over NFS (I use NFS root on my devices) but eventually I got git with all dependencies up and running.

!!! Please note that packages are not officially provided by FreeBSD Project. They’re only for experimental purpose so install them at your own discretion !!!

Thanks to bapt@ for working on this great tool.

* – some
** – And for other ARM devices

VCHI driver, part 2

January 8, 2013 — Leave a comment

Some time ago I announced port of VCHI driver to FreeBSD. Since then it was re-licensed as BSD/GPL and I had high hopes for bringing it into the tree as a part of sys/contrib. This weekend I finally got around to it but turned out things had changed for worse. VCHI driver used to have this neat OS-abstraction wrapper, so overall porting process was quite simple: implement synch primitives, physical pages management, driver-specific initialization and you’re done. But… The layer was lost with driver update in October. The reason for it – OS compatibility shims are banned from Linux mainline kernel.

So now I face two options: create complete port of VCHI driver by replacing linux-specific parts with freebsd-specific code. Or create Linux compatibility shims and try to keep sources as close as possible to upstream. I’m going to try latter approach in order to minimize maintenance work. If it doesn’t work out – I’ll fall back to the former. But with any of these approaches code difference with upstream will be too significant to go to contrib tree and most likely driver will be distributed as a port.