Cross Building FreeBSD

I've been getting more and more questions about FreeBSD embedded systems: What is supported? How can I build for them? What can be done to improve the cross development environment? How can I help? and so on. These are all very good questions, so I thought I'd start a series of articles on the FreeBSD build system and embedding FreeBSD into products.

For this article, I'll be providing background on the FreeBSD build system as it relates to embedded systems. For an emebedded system, one usually has little more than a microcontroller with an MMU to power the entire system. While these systems work well in their target niche, they don't work so well as a general purpose platform. They lack the memory and I/O bandwidth necessary to compile applications that run on them. Server machines (or even workstations) do a much better job at these tasks, but are for a different computer architecture. One is left with the choice of compiling natively on a slow machine, or cross building on a fast machine. I'll discuss some basic aspects of cross compilation in this article.

I'll assume that you are familiar with FreeBSD's build system. If not, there are many resources available on the web. One of the better ones is the FreeBSD handbook. Future articles will build on this base, hopefully with sufficient links.

Each FreeBSD platform has a MACHINE and a MACHINE_ARCH that uniquely define it. The MACHINE_ARCH is the CPU family that executes the object code. MACHINE_ARCH is something like "i386" or "sparc64." Some CPU families come in different flavors that are mutually incompatible (such as MIPS processors using different byte ordering). These families would get a different MACHINE_ARCH for each flavor (so you'd have mipseb and mipsel, for example, to distinguish the two different flavors of MIPS CPUs). If you are familar with the gnu compilers and such, this field roughly corresponds to "CPU" field of their "CPU-MANUFACTURER-OS" identifier. I say "corresponds" because FreeBSD uses "amd64" as the MACHINE_ARCH for AMD's extensions to the x86 architecture (the official name of the architecture), while gcc uses the older x86_64 designation.

MACHINE defines the way the machine is put together. Often times machines are build with the same CPU, but with differing glue chips and support software. In cases like this, if the machines are very similar, FreeBSD will support that with one MACHINE type. In other cases, the differences are large enough to warrant an entirely separate MACHINE for that machine. What differentiates one MACHINE type from another with the same MACHINE_ARCH are things like BIOS interface, boot procedure, disk partitioning, and expansion bus technologies. For example, the FreeBSD/i386 and FreeBSD/pc98 platforms both execute on i386 compatible CPUs, but they have completely different bus topologies, boot procedures, BIOS interfaces, and disk layout.

When one builds FreeBSD on one platform to execute on another platform, that is called cross compiling. Cross compilation support is integrated into the normal FreeBSD build system. It is activated by defining TARGET and TARGET_ARCH of the machine you are targeting and using the normal make targets. For example:

make TARGET=arm TARGET_ARCH=arm buildworld

will build the entire userland portion of the system for an ARM machine. Similarly, the following:

make TARGET=arm TARGET_ARCH=arm installworld DESTDIR=/foo

will install these components into the directory tree /foo. One can even build and install a kernel for the target platform:

make TARGET=arm TARGET_ARCH=arm KERNCONF=BONGO kernel DESTDIR=/foo

After cross compilation support was added to the tree, the FreeBSD project added two additional make targets which allow use of the build tools outside of the build tree. The first is "buildenv." It forks a shell with the same build environment used to build FreeBSD. This target is for interactive development. Much of the Atmel AT91RM9200 port was done using this target, for example. However, there's no ability to specify a command to run in this sub-shell, nor does it lend itself well to automated building systems. The make target "buildenvvar" was created to try to fill the gap in the latter problem. This target prints the build environment variables used by the build system. We used it at my day joy to help automate building our products on an x86 processor for an embedded ARM design we're producing. This works well for many applications, but not for all. We have found that we can make it work for simple applications, but more complicated one, with more elaberate build systems, require something more sophisticated. In a future article, I'll describe how we solved that problem.

Many times, cross building can be useful for "compile testing" changes to the tree. There's a make target called "universe" that builds the entire tree for each of the Tier 1 and 2 architectures, along with all the kernels that it can find. In many cases, these "compile tests" of the system find problems that would otherwise be found by others, thus saving much embarassment. The drawback, however, to a "universe" build is that it takes several hours on a very fast machine.

I hope that you've found the above information useful, and that it answers your basic questions in this area. Feel free to leave feedback with additional questions or requests for clarification and I'll see what I can do about addressing them in the feedback section, or in future articles.