Hard Float API coming soon by default to armv6

All the CPUs that FreeBSD supports have hard floating point in them. We've supported hard float for quite some time in the FreeBSD kernel. However, by default, we still use a soft-float ABI. All the binaries use hardware floating point to do the math, but the calling conventions between routines passes the floating point values in integer registers to remain compatible with the soft-float libraries of the past. A new armv6hf was created, but that caused some issues with some ports, and the meaning of 'soft float' sadly was ambiguous between the soft-float ABI, and the soft-float libraries that implement floating point when there's no hardware FPU.

Over the spring and summer, I fixed ld.so so that it  can load both soft ABI and hard ABI libraries on the same system, depending markings in the binaries themselves. Soft float ABI and hard float ABI binaries have different flags in the ELF headers, so it is relatively straight forward to know which is which. Over the summer, I committed changes to the kernel to pass the relevant flags from the header to user land (since for a variety of technical reasons, once you get to user land, it's impossible to open the binary you are executing in the general case). I had planned on committing the ld.so bits as well, but my job got busy and now it's winter.

So, in the coming days, I'll commit the first set of changes to move to armv6 as a hard float ABI by default. The kernel doesn't care: it can execute both. The new ld.so will allow you to transition through this change by allowing old, compat soft ABI libraries to co-exist on the system with new hard ABI libraries. This change alone isn't enough, but it will be good to get it out into circulation.

After that, a number of changes will follow. Similar to the LIB32 stuff for x86, mips and powerpc, there will be a LIBSOFT build that you can enable on armv6. This will create the libraries needed for the transition, and will be used if we need to generate compat binaries for this stuff (the hope is that the number of old binaries that need to run on the new system will be small enough that it can be treated as a special-needs case, not one that's needed by default). The normal armv6 build will switch to hard float ABI by default. armv6hf will remain in the build as a compatibility arch, and will be removed before FreeBSD 11 since it was never in FreeBSD 10 (and could never be in FreeBSD 10 due to compiler choices). TARGET_CPUTYPE will grow a new type or two for those folks with really special needs that need to generate wither soft-float libraries, or soft-float ABI libraries (and this will be the mechanism used to support LIBSOFT builds).

NanoBSD reved up for embedded

NanoBSD Enhancements

I've been experimenting with NanoBSD enhancements for embedded. Eventually, they will be in the mainline of NanoBSD, but for now I've segregated them in a separate embedded subdir. The goal is to drive creation of build-agnostic tools to help build images for the different systems that FreeBSD supports. I have a mix of embedded boards and qemu configurations. Many don't work, but I hit a milestone tonight. I was able to boot an image on my RPi built without privs using NanoBSD.

To try it out, you'll need to grab FreeBSD-current and from the top of the tree do

cd tools/tools/nanobsd/embedded

sh ../nanobsd.sh -c rpi.cfg

This will create a directory at the same level as your top level FreeBSD source called rpi-b. Under it will be obj/_.disk.image.rpi-b. You can dd this onto a SD card (any size) and boot it on your RPi-B. It will resize the last partition to span the rest of the disk on boot and be a generic system. There's still a few problems, and I'm still working out how to get packages onto the system at build time, but its useful enough that people may want to play with it. Enjoy.

P.S. Here's a dmesg from the boot:

U-Boot 2015.10 (Nov 09 2015 - 02:54:46 +0000)

DRAM:  480 MiB
RPI Model B rev2
MMC:   bcm2835_sdhci: 0
reading uboot.env

** Unable to read "uboot.env" from mmc0:1 **
Using default environment

In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
reading uEnv.txt
0 bytes read in 9 ms (0 Bytes/s)
Hit any key to stop autoboot:  0 
starting USB...
USB0:   Core Release: 2.80a
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
       scanning usb for ethernet devices... 1 Ethernet Device(s) found
Booting from: mmc 0 ubldr.bin
reading ubldr.bin
214684 bytes read in 36 ms (5.7 MiB/s)
## No elf image at address 0x00200000
## Starting application at 0x00200000 ...
Consoles: U-Boot console  
Compatible U-Boot API signature found @1db474d0

FreeBSD/armv6 U-Boot loader, Revision 1.2
(imp@zooty, Wed Dec 23 17:23:10 MST 2015)

DRAM: 480MB
Number of U-Boot devices: 2
U-Boot env: loaderdev='mmc 0'
Found U-Boot device: disk
  Checking unit=0 slice= partition=... good.
Booting from disk0s3a:
/boot/kernel/kernel text=0x587d2c data=0x54f24+0xe639c syms=[0x4+0xc6460+0x4+0x94335]

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by U-Boot at address 0x100.
Kernel entry at 0x400180...
Kernel args: (null)
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r292645 0ff2d15e64f9+ qbase/qtip/rpi/tip: Wed Dec 23 17:29:02 MST 2015
    imp@zooty:/usr/home/imp/FreeBSD/rpi-b/obj/arm.armv6/usr/home/imp/FreeBSD/head/sys/RPI-B arm
FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906
VT: init without driver.
sema_sysinit
CPU: ARM1176JZ-S rev 7 (ARM11J core)
 Supported features: ARM_ISA THUMB2 JAZELLE ARMv4 Security_Ext
 WB enabled LABT branch prediction enabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 503312384 (479 MB)
avail memory = 481800192 (459 MB)
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: 
simplebus0:  mem 0x20000000-0x20ffffff on ofwbus0
cpulist0:  on ofwbus0
cpu0:  on cpulist0
bcm2835_cpufreq0:  on cpu0
intc0:  mem 0xb200-0xb3ff on simplebus0
systimer0:  mem 0x3000-0x3fff irq 8,9,10,11 on simplebus0
Event timer "BCM2835-3" frequency 1000000 Hz quality 1000
Timecounter "BCM2835-3" frequency 1000000 Hz quality 1000
bcmwd0:  mem 0x10001c-0x100027 on simplebus0
gpio0:  mem 0x200000-0x2000af irq 57,59,58,60 on simplebus0
gpio0: read-only pins: 46-53.
gpio0: reserved pins: 48-53.
gpiobus0:  on gpio0
gpioled0:  at pin 16 on gpiobus0
gpioc0:  on gpio0
iichb0:  mem 0x205000-0x20501f irq 61 on simplebus0
iicbus0:  on iichb0
iic0:  on iicbus0
iichb1:  mem 0x804000-0x80401f irq 61 on simplebus0
iicbus1:  on iichb1
iic1:  on iicbus1
spi0:  mem 0x204000-0x20401f irq 62 on simplebus0
spibus0:  on spi0
bcm_dma0:  mem 0x7000-0x7fff,0xe05000-0xe05fff irq 24,25,26,27,28,29,30,31,32,33,34,35,36 on simplebus0
mbox0:  mem 0xb880-0xb8bf irq 1 on simplebus0
sdhci_bcm0:  mem 0x300000-0x3000ff irq 70 on simplebus0
mmc0:  on sdhci_bcm0
uart0:  mem 0x201000-0x201fff irq 65 on simplebus0
uart0: console (115200,n,8,1)
vchiq0:  mem 0xb800-0xb84f irq 2 on simplebus0
vchiq: local ver 8 (min 3), remote ver 8.
pcm0:  on vchiq0
bcm283x_dwcotg0:  mem 0x980000-0x99ffff irq 17 on simplebus0
usbus0 on bcm283x_dwcotg0
fb0:  on ofwbus0
fbd0 on fb0
VT: initialize with new VT driver "fb".
fb0: 656x416(656x416@0,0) 24bpp
fb0: fbswap: 0, pitch 1968, base 0x1e08c000, screen_size 818688
cryptosoft0: 
Timecounters tick every 10.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 480Mbps High Speed USB v2.0
bcm2835_cpufreq0: ARM 700MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF
ugen0.1:  at usbus0
uhub0:  on usbus0
mmc0: CMD8 failed, RESULT: 1
mmcsd0: 513MB  at mmc0 41.6MHz/4bit/65535-block
Trying to mount root from ufs:/dev/mmcsd0s3a [ro]...
warning: no time-of-day clock registered, system time will not be set accurately
uhub0: 1 port with 1 removable, self powered
ugen0.2:  at usbus0
uhub1:  on usbus0
uhub1: MTT enabled
uhub1: 3 ports with 2 removable, self powered
random: unblocking device.
ugen0.3:  at usbus0
smsc0:  on usbus0
smsc0: chip 0xec00, rev. 0002
miibus0:  on smsc0
ukphy0:  PHY 1 on miibus0
ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ue0:  on smsc0
ue0: Ethernet address: b8:27:eb:f1:62:e6
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Growing root partition to fill device
GEOM_PART: mmcsd0s3 was automatically resized.
  Use `gpart commit mmcsd0s3` to save changes or `gpart undo mmcsd0s3` to revert them.
mmcsd0s3 resized
mmcsd0s3a resized
super-block backups (for fsck_ffs -b #) at:
 556192, 667424, 778656
/etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Setting hostuuid: 901c1969-a9d5-11e5-996d-b827ebf162e6.
Setting hostid: 0x45acda89.
No suitable dump device was found.
Starting file system checks:
/dev/mmcsd0s3a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mmcsd0s3a: clean, 181490 free (26 frags, 22683 blocks, 0.0% fragmentation)
/dev/mmcsd0s2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mmcsd0s2: clean, 32726 free (6 frags, 4090 blocks, 0.0% fragmentation)
Mounting local file systems:.
Setting hostname: nanobsd-rpi-b.
Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy:eval: cannot create /boot/entropy: Read-only file system
smsc0: chip 0xec00, rev. 0002
ue0: link state changed to DOWN
ue0: link state changed to UP
Starting Network: lo0 ue0.
lo0: flags=8049 metric 0 mtu 16384
 options=600003
 inet6 ::1 prefixlen 128 
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
 inet 127.0.0.1 netmask 0xff000000 
 groups: lo 
 nd6 options=21
ue0: flags=8843 metric 0 mtu 1500
 options=80009
 ether b8:27:eb:f1:62:e6
 media: Ethernet autoselect (100baseTX )
 status: active
 nd6 options=29
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Starting devd.
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Starting dhclient.
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5
DHCPOFFER from 10.0.0.5
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 10.0.0.5
bound to 10.0.0.65 -- renewal in 3600 seconds.
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Generating host.conf.
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Creating and/or trimming log files.
Starting syslogd.
Setting date via ntp.
24 Dec 01:06:34 ntpdate[457]: step time server 204.9.54.119 offset 2122.146331 sec
Starting casperd.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Configuring vt: blanktime.
Starting cron.
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Starting background file system checks in 60 seconds.
override rw-rw-r--  730/730 for /firstboot? 

Thu Dec 24 01:06:47 UTC 2015

FreeBSD/arm (nanobsd-rpi-b) (ttyu0)

login: root
Dec 24 01:06:50 nanobsd-rpi-b login: ROOT LOGIN (root) ON ttyu0
FreeBSD 11.0-CURRENT (RPI-B) #0 r292645 0ff2d15e64f9+ qbase/qtip/rpi/tip: Wed Dec 23 17:29:02 MST 2015

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
root@nanobsd-rpi-b:~ # mount -uw /
root@nanobsd-rpi-b:~ # rm /firstboot
root@nanobsd-rpi-b:~ # uname -a
FreeBSD nanobsd-rpi-b 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r292645 0ff2d15e64f9+ qbase/qtip/rpi/tip: Wed Dec 23 17:29:02 MST 2015     imp@zooty:/usr/home/imp/FreeBSD/rpi-b/obj/arm.armv6/usr/home/imp/FreeBSD/head/sys/RPI-B  arm
root@nanobsd-rpi-b:~ # uptime
 1:07AM  up 1 min, 1 users, load averages: 0.87, 0.39, 0.16
root@nanobsd-rpi-b:~ # ifconfig
lo0: flags=8049 metric 0 mtu 16384
        options=600003
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21
ue0: flags=8843 metric 0 mtu 1500
        options=80009
        ether b8:27:eb:f1:62:e6
        inet 10.0.0.65 netmask 0xffffff00 broadcast 10.0.0.255 
        media: Ethernet autoselect (100baseTX )
        status: active
        nd6 options=29
root@nanobsd-rpi-b:~ # ntpdate 0.freebsd.pool.ntp.org
load: 0.48  cmd: ntpdate 671 [select] 8.16r 0.05u 0.04s 0% 3112k
24 Dec 01:07:44 ntpdate[671]: adjust time server 69.164.201.165 offset 0.001943 sec
root@nanobsd-rpi-b:~ # exit
logout

FreeBSD/arm (nanobsd-rpi-b) (ttyu0)

login: 

Rearranging the deck chairs in the ARM port

Recently, I've been trying to catch up with some of the technical debt that the FreeBSD/arm port has accumulated.  Some of that technical debt was my fault, of course, but some of it wasn't.  I don't really much care whose fault things were,  but I would like to get things cleaned up.  Some of these are paths not taken.  Some are porting shims that never got properly connected.  There's also some features that were poo-poo-ed by some (including me) years ago that seem to make more sense now.

None of this stuff is terribly sexy.  Some of it will be addressed by the summer of code project that's working its way through some of the maze of twisty passages in initarm() that are subtly different between the different boards for no good reason.

We have no common place for boot loader code.  Some arm ports do use a common routine to fake up just enough metadata that debugging and other symbols work.  Some arm ports assume a full /boot/loader is present.  However, there's no common facility for getting information from via the Linux boot protocol, nor is there any provisions for having multiple boards supported by a single kernel.  There's no uniform interface to get this information that might be passed in by other means.  There's no real way for a board to get control early enough to get at meta-data a custom boot loader might pass in.

The first big thing that I'd like to comment on is a small change to the interface between locore.S' _start routine and the first port-specific code in initarm.  I've preserved the first 4 registers that were passed in on boot to _start().  Before, there'd be no way to access the Linux ATAGs, for example, by a port since these are passed in as arg 3.  To allow for future expansion, I'm just passing in one structure now.

In the coming weeks, I'll be implementing routines to parse these arguments based on standard boot protocols, while allowing customization for special needs.

In the coming months, I plan to expand our support for having multiple boards based on a single Atmel SoC.  Once I have the one at a time SoC code working, I hope to get multiple SoCs working in one kernel.  While the extra 'bloat' isn't optimal for many users, the convenience of being able to boot one kernel for installation, or trying out the system, is a feature that's been much requested.  I've already eliminated the need to have a compiled-in master clock frequency, with improved main clock detection with fallback for people using unconventional frequencies.

I'm also interested in sweeping into the tree many of the Atmel arm changes that have been accumulating in the PR database.  Some of these will be easy to integrate, while others may be difficult due to drift from the original submission.

Once we get a good baseline, I hope to merge these changes into 9.x and 8.x.  To date, the interfaces that have change have been purely internal ones we do not support.  External users may experience some bumpy waves if they haven't been working to get their changes merged upstream.  The more that you've submitted, the more I'm likely to go the extra mile to help smooth any transitions.  With luck, 8.3 and 9.2 will both benefit from these efforts.

What is all the TBEMD stuff

I've been getting questions about what TBEMD means in my commits to FreeBSD head.

First, I'll tell you the name. TBE stands for TARGET_BIG_ENDIAN. For the MIPS and ARM ports, it is an environment variable that you have to set to build these targets for big endian (otherwise they default to little endian). MD is Must Die. A few years ago, there was an action movie whose title was more memorable than the file "Romeo Must Die." So I just stole the name for this branch.

So why must TARGET_BIG_ENDIAN die? It breaks things. For all the other platforms that FreeBSD supports, you just set TARGET and/or TARGET_ARCH to do cross builds. You know everything you need to know from the MACHINE and MACHINE_ARCH in the resulting image. In addition, the build system segregates things so you can build all the targets in one tree..But for mips and arm, you don't know what endian a binary is, the obj tree will collide if you try to build both little endian and big endian binaries at the same time. There's also other, more subtle issues. For example, TARGET_BIG_ENDIAN isn't set on big endian mips, so a native biuldworld tries to build little endian binaries (oops).

So, mips will be moving to mipsel (for little endian) and mipseb (for big endian). Arm will move to arm (for little endian) and armeb (for big endian). You could fill an entire email archive with all the possible other names and why they are better or worse than these names. History, however, trumps all those arguments: these are the names used elsewhere and we're just following convention.

Once tbemd is completely merged, you'll be able to build both endians of MIPS in the same object tree, for example, and all the other issues I've discovered. It also helps with new architectures as we move into powerpc64 (already merged) and mips64 (to be done after tbemd is collapsed). It will also mean we can have different packages for the different endians now.

Moving towards board files

I've spent a little bit of time implementing the start of board files for the arm port. The initial push has been for the at91 subport only, and many improvements could be made to this. I've written up my initial thoughts on this on the FreeBSD wiki FreeBSD Arm Boards. It could use much improvement, I'm sure.

One idea that hasn't been reflected there yet, was shown to me by Sam Laffler who suggested using linker sets to allow boards to 'probe', 'init' and other standardized functions. This is an interesting idea and I plan on working on adding it to the above links when Sam has results to share.

I'd also like to expand the above wiki page to be a 'best practices' guide for all architectures where there's great diversity of boards/cpus/etc (eg, not the homogeneous env that x86 provides).

I'm also soliciting comments on the above boards in addition to the above. Send them to me, or post them here.

Marvell ARM support in svn

Rafal Jaworowski just committed support for a number of Marvell Processors. There are a number of commits going into the tree. He's just completed the first of these:
Introduce low-level support for new Marvell core CPUs: 88FR131, 88FR571.

From an earlier email message, we know that Rafal is working on support for Marvell 88F5182, 88F5281, 88F6281, and MV78100 SoCs.

The following peripherals will likely be supported:

• EHCI USB 2.0
• Ethernet
• GPIO
• Interrupt controller
• L1, L2 cache
• Timers, watchdog, RTC
• TWSI (I2C)
• UART

This is a fairly mature port that is going into the tree. It works with NFS or USB mounted root filesystems. The port is self hosted. Kernel and world builds succeed on the box. The box also boots to multiuser mode. Rafal has created a web page with all this information on it on the FreeBSD wiki at http://www.wiki.freebsd.org/FreeBSDMarvell

This is way cool news! Good Job Rafal.

Note: Much of this information was taken from a posting Rafal made to arm@freebsd.org.

FreeBSD embedded (mips, powerpc, arm) update

The last week or so has show a spike up in FreeBSD embedded platform work. Fixes for mips and powerpc have gone into the tree. Support for a new ARM platform is getting ready to be committed. FreeBSD/mips is almost self-hosting in the svn tree (it has been in the p4 tree for some time). FreeBSD/powerpc and FreeBSD/arm continue to mature.

Thanks to a flurry of activity over the weekend by David O'Brien, the number of patches needed for FreeBSD/mips has been reduced to a set for libpam and for binutils. David cleaned up the diffs for gcc and pushed them into the tree in a manner that allows for easier upstream committing.

In addition, David has cleaned up a number of dangling nits in the build. This is the result of a slight mismatch between p4 and the code in FreeBSD's svn tree. Oleksandr Tymoshenko has looped back most of these changes into p4, and is working on resolving the conflicts that were generated. Once this is sorted out, Oleksandr or myself will generate a new set of diffs and go for the final push into the tree.

I'm looking forward to having a good base for FreeBSD/mips in the tree. I've been busy with a number of other projects and haven't given it the time that I really want to give.

On the PowerPC side of the ledger, Marcell Moolenaar and Rafal Jaworowski have been polishing uboot boot loader support in the FreeBSD/powerpc subversion tree. In addition, some bug fixes to and code polishing of the low level Book E PowerPC support have been committed. Many of these improvements were obtained from Semihalf consulting.

On deck to be committed soon is support for Marvell's Orion NAS chips. This code has been in perforce for a while now and is nearly ready for the FreeBSD tree. I'll keep everybody posted when that happens. I hope to have my DLINK DIR-615 Rev A1 working by then with this code, but time has been tight and there's already too many different projects afoot in my life.

Slimming down the NSLU kernel

A few days ago, I wrote about the NSLU kernel config file being committed to FreeBSD's svn tree. The default compressed kernel is about 1.6MB, but the size of the partition of the NSLU2's flash is 1.25MB, a gap of just under 300kB. With some hacking, I've been able to reduce the size of the kernel to fit.

The default kernel's text+data size is 3,425,844 bytes. Compressed, with the appropriate headers, this drops to 1,591,109 bytes. However, the goal is 1,310,720 bytes. We're short by at least 294,553 bytes of fitting into the kernel portion of the flash on the NSLU.

My first thought was to start cutting things out. I removed IPv6 (options INET6) since I didn't need it for my setup. This saved 210,797 bytes in the uncompressed kernel and 74,115 bytes in the compressed kernel. This one option shaved 5% (compressed) of the kernel size. A move in the right direction, but more was needed.

Next, I recalled from previous attempts that there's many inline functions that don't need to be inlined. There's a small performance gain from inlining these functions, but it can add a lot to the kernel. So I added three options to the kernel config:

options MUTEX_NOINLINE
options RWLOCK_NOINLINE
options SX_NOINLINE

This resulted in a size reduction of 432,984 bytes (uncompressed) and 87,289 bytes (compressed). This is also a significant savings, and meant that we'd saved 161,404 bytes in the compressed kernel so far. We were still ~133,000 bytes shy of the mark.

At this point I noticed that there were a lot of things in the kernel that weren't required for boot, and that had shown up as being larger. When doing the size reduction, often times I use the 'size' command to get the top 20 .o files to see if there's junk that can just be omitted from the kernel. In this case, I saw that usb, scsi and releated drivers consumed a lot of space. This could be loaded from a number of modules after boot if I was booting from a RAM disk. Since the plan was to put an initrd-like ram disk in the other partition of the flash, I went ahead and removed these pieces:

#device usb
#options USB_DEBUG
#device ohci
#device ehci
#device ugen
#device umass
#device scbus # SCSI bus (required for SCSI)
#device da # Direct Access (disks)

This trimming saved 278,837 bytes (uncompressed) and 133,520 bytes (compressed). This was huge, and brought the total trimmed up to 294,924 bytes. This is just barely over the line, so I thought I'd go for a little more.

Another big area of the kernel is the code brought in with 'options miibus' configuration line. This brings in all the phy drivers on the off chance you'll need it. In many embedded systems, you know which phy driver you have, and can wire in only the one or ones you need. Since I wasn't planning on using this box as a general purpose router, that needed to work with any Ethernet dongle that was plugged into the USB port, I went ahead and transitioned to this configuration. The NSLU has a RealTek Phy that needs the rlphy driver. This means that I changed the config file like so:

#device miibus # NB: required by npe
device mii
device rlphy

This option shaved a few more bytes off the total. 80,012 bytes (uncompressed) and 25,156 bytes (compressed).

At the end of all this, the compressed kernel had been reduced from 1,625,125 bytes down to 1,286,193 bytes, for a savings of 320,080 bytes. This was enough margin for me to feel comfortable trying to burn it into my NSLU. Unfortunately, I ran out of time tonight to try it, so the story of that experiment will wait for another post. Also, I don't know how to make blogspot do tables, or I'd have made some nice tables with all this data for this log.

There's likely more options that can be explored to squeeze even more bytes out of the kernel. bzip2 might be a good place to get maybe 10% more space back (at these sizes, an extra 100kB compressed, or 250kB uncompressed). The exploder for bzip2 is bigger and uses more memory.

Looking at the .o files that are laying around that pci.o is kinda big, suggesting that we need to break it up some more. In previous experiments, separating out the MSI and PCIe stuff helped quite a bit. I'm unsure what the best newbusly way to extract this functionality might be, but it could save us 20kB uncompressed (maybe 8k compressed, iirc).

There's a number of other places code is agresively inlined, these could be investigated. there's
a few things in the tree tagged as standard that likely could be made to be options. There's much room here to explore a number of different options to reduce the kernel size even further if the need presents itself.

The nfs server option is about 45kB. Soft updates run about the same. nfsv4 also eats a lot of memory (looks to be 55kB).

However, for all these options, it would be nice is the NAS could have RAID, FAT, ntfs, smbfs and ext2fs support as well. Continuing to trim would allow this more easily. Or at least eliminate the need for loadable modules.

This is to say, there's a lot of areas for further research and documentation. I didn't even look at using a /boot/loader either. Maybe that can be helpful... Maybe I'll turn that into a research paper for one of the conferences... Either the continued size reduction thing, or using /boot/loader.

Finally, I've uploaded the changed NSLU config file in case my verbal descriptions here were insufficient.

NSLU2 support committed to FreeBSD/arm

I just committed the basic Linksys NSLU2 support to the subversion tree. It boots and runs on an NFS root. We're currently lacking support for the leds, the buttons and the on-board flash. Here's some basic instructions if you want to try it on your NSLU2. They are adapted from Sam Leffler's README file for the AVILA, which is adapted from other sources, including Olivier Houchard's and my instructions that have appeared.

1. Build world
2. Setup NFS root
3. Configure diskless system
4. Build and Install Kernel
5. Put Kernel in tftp area
6. Boot from redboot

Build world

The FreeBSD handbook can help you find the sources, etc Once you have them, and they are from -CURRENT after 8am on 2008-08-03, you can proceed to build the world. Each section is cumulative from the previous section for simplicity sake.

setenv TARGET arm
setenv MAKEOBJDIRPREFIX /tmp/$USER/obj
setenv TARGET_BIG_ENDIAN t
setenv TARGET_CPU_TYPE xscale
make buildworld

Setup NFS root

Find a server that has some disk space. You'll need about 250MB of space for a full FreeBSD image.

setenv ROOT /data/freebsd/roots/slug
make installworld DESTDIR=$ROOT
mergemaster -m $SRC/etc -D $ROOT -i -A arm

Setup diskless system
diskless(8) gives a good overview in how to do this. It is left as an exercise for the read to come up with all the settings. Usually /etc/rc.conf and /etc/fstab must be configured. /etc/ttys is another one that usually needs some tlc. If there's enough demand, I'll write about this in the future.

Build and install the kernel

make KERNCONF=NSLU buildkernel
make KERNCONF=NSLU DESTDIR=$ROOT installkernel

Note: the NSLU kernel is configured with the root filesystem mounted via NFS over the npe0 wired interface.

You should now have an NFS-mountable root filesystem with a kernel. The final step is to setup network diskless booting from the board. It is assumed you have a DHCP server operating on your network and the server is configured to supply the necessary information in the DHCP lease. If you run the ISC DHCP server the following configuration information is an example of how to do this:

option root-opts code 130 = string; # NFS / mount options

host slug {
hardware ethernet 00:d0:12:02:47:68;
fixed-address 10.0.0.221;
next-server 10.0.0.251;
filename "kernel-slug.nfs";
option root-path "10.0.0.251:/data/freebsd/roots/slug";
option root-opts "nolockd";
}

Note: the root-opts item specifies the root filesystem should be mounted with the nolocked option; this just short-circuits file locking requests so you don't get complaints from programs that use the pidfile(3) routine (e.g. devd)

Install kernel in tftp area

Place the kernel in the TFTP area for booting from the prom monitor. If your TFTP server returns file from /tftpboot (default) then do something like:

cp $ROOT/boot/kernel/kernel /tftpboot/kernel-slug.nfs

Boot using Redboot

+(hit control C here)
RedBoot> ip -h 10.0.0.251 -l 10.0.0.1
IP: 10.0.0.1/255.255.255.0 Gateway: 0.0.0.0
Default server: 10.0.0.251
RedBoot> load -b 0x200000 kernel-avila.nfs
Using default protocol (TFTP)
Address offset = 0x40000000
Entry point: 0x00200100, address range: 0x00200000-0x004db2d4
RedBoot> go

dmesg
Here's a complete transcript of my doing this. The .gz file listed below really isn't the simple gz file, but rather kernel.gz.tramp from the kernel build. Also, I've hacked my NSLU to have a serial port, which make development like this easier. See the web for instructions on how to do a no-mod breakin.

+Ethernet eth0: MAC address 00:14:bf:68:47:ba
IP: 192.168.0.1/255.255.255.0, Gateway: 192.168.0.1
Default server: 0.0.0.0, DNS server IP: 0.0.0.0

RedBoot(tm) bootstrap and debug environment [ROMRAM]
Red Hat certified release, version 1.92 - built 15:16:07, Feb 3 2004

Platform: IXDP425 Development Platform (XScale)
Copyright (C) 2000, 2001, 2002, Red Hat, Inc.

RAM: 0x00000000-0x02000000, 0x000723a0-0x01ff3000 available
FLASH: 0x50000000 - 0x50800000, 64 blocks of 0x00020000 bytes each.
== Executing boot script in 2.000 seconds - enter ^C to abort
^C
RedBoot> ip -l 10.0.0.12 -h 10.0.0.8
IP: 10.0.0.12/255.255.255.0, Gateway: 192.168.0.1
Default server: 10.0.0.8, DNS server IP: 0.0.0.0
RedBoot> load -b 0x200000 kernel.nslu.gz
Using default protocol (TFTP)
Address offset = 0xf0000000
Entry point: 0x00200074, address range: 0x00200000-0x003830fc
RedBoot> go
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #12: Sun Aug 3 00:09:07 MDT 2008
imp@lighthouse:/tmp/imp/obj/arm/cache/svn/head/sys/NSLU
CPU: IXP425 266MHz rev 1 (ARMv5TE) (XScale core)
DC enabled IC enabled WB enabled LABT branch prediction enabled
32KB/32B 32-way Instruction cache
32KB/32B 32-way write-back-locking Data cache
real memory = 33554432 (32 MB)
avail memory = 25894912 (24 MB)
ixp0: on motherboard
pcib0: on ixp0
pci0: on pcib0
ohci0: irq 28 at device 1.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0
usb0: on ohci0
usb0: USB revision 1.0
usbd_get_string: getting lang failed, using 0
uhub0: on usb0
uhub0: 3 ports with 3 removable, self powered
ohci1: irq 27 at device 1.1 on pci0
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb1: OHCI version 1.0
usb1: on ohci1
usb1: USB revision 1.0
usbd_get_string: getting lang failed, using 0
uhub1: on usb1
uhub1: 2 ports with 2 removable, self powered
ehci0: irq 26 at device 1.2 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb2: EHCI version 1.0
usb2: companion controllers, 3 ports each: usb0 usb1
usb2: on ehci0
usb2: USB revision 2.0
uhub2: on usb2
uhub2: 5 ports with 5 removable, self powered
ixpclk0: on ixp0
ixpiic0: on ixp0
iicbb0: on ixpiic0
iicbus0: on iicbb0 master-only
iicbus0: at addr 0
iic0: on iicbus0
ixpwdog0: on ixp0
uart0: on ixp0
uart0: [FILTER]
uart0: console (115200,n,8,1)
uart1: on ixp0
uart1: [FILTER]
ixpqmgr0: on ixp0
ixpqmgr0: [ITHREAD]
npe0: on ixp0
npe0: [ITHREAD]
npe0: remember to fix rx q setup
miibus0: on npe0
rlphy0: PHY 1 on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
npe0: Ethernet address: 00:14:bf:68:47:ba
npe1: on ixp0
npe1: [ITHREAD]
npe1: remember to fix rx q setup
npe1: Cannot find my PHY.
device_attach: npe1 attach returned 6
ixpclk0: [FILTER]
Timecounter "IXP425 Timer" frequency 66666600 Hz quality 1000
Timecounters tick every 10.000 msec
bootpc_init: wired to interface 'npe0'
Sending DHCP Discover packet from interface npe0 (00:14:bf:68:47:ba)
npe0: link state changed to UP
Received DHCP Offer packet on npe0 from 0.0.0.0 (accepted) (no root path)
Sending DHCP Request packet from interface npe0 (00:14:bf:68:47:ba)
Received DHCP Ack packet on npe0 from 0.0.0.0 (accepted) (got root path)
npe0 at 10.0.0.12 server 0.0.0.0 boot file kernel.slug
subnet mask 255.255.255.0 router 10.0.0.1 rootfs 10.0.0.8:/pe/slug rootopts nolockd
Adjusted interface npe0
Trying to mount root from nfs:
NFS ROOT: 10.0.0.8:/pe/slug
warning: no time-of-day clock registered, system time will not be set accurately
warning: no time-of-day clock registered, system time will not be set accurately
Interface npe0 IP-Address 10.0.0.12 Broadcast 10.0.0.255
Loading configuration files.
No suitable dump device was found.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
mount_nfs: can't update /var/db/mounttab for 10.0.0.8:/pe/slug
Setting hostuuid: 1c728616-1dd2-11b2-9fa6-0014bf6847ba.
Setting hostid: 0xb1c1273a.
Mounting local file systems:.
Setting hostname: slug.
net.inet6.ip6.auto_linklocal: 1 -> 0
DHCPREQUEST on npe0 to 255.255.255.255 port 67
ip length 328 disagrees with bytes received 332.
accepting packet with data after udp payload.
DHCPACK from 10.0.0.1
bound to 10.0.0.12 -- renewal in 900 seconds.
lo0: flags=8049 metric 0 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
npe0: flags=8843 metric 0 mtu 1500
ether 00:14:bf:68:47:ba
inet6 fe80::214:bfff:fe68:47ba%npe0 prefixlen 64 scopeid 0x1
inet 10.0.0.12 netmask 0xffffff00 broadcast 10.0.0.255
media: Ethernet autoselect (100baseTX )
status: active
Additional routing options:.
Starting devd.
Additional IP options:.
Mounting NFS file systems:.
Creating and/or trimming log files:.
Starting syslogd.
/etc/rc: WARNING: Dump device does not exist. Savecore not run.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Initial arm initialization:.
Additional ABI support:.
Clearing /tmp (X related).
Starting local daemons:.
Updating motd.
Mounting late file systems:.
Starting cron.
Local package initialization:.
Starting inetd.
Starting background file system checks in 60 seconds.

Thu Jan 1 00:00:39 UTC 1970

FreeBSD/arm (slug) (ttyu0)

login: root
Jan 1 00:03:24 slug login: ROOT LOGIN (root) ON ttyu0
Last login: Thu Jan 1 00:42:28 on ttyu0
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 8.0-CURRENT (NSLU) #12: Sun Aug 3 00:09:07 MDT 2008

Welcome to FreeBSD!

Before seeking technical support, please use the following resources:

o Security advisories and updated errata information for all releases are
at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
for your release first as it's updated frequently.

o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
along with the mailing lists, can be searched by going to
http://www.FreeBSD.org/search/. If the doc distribution has
been installed, they're also available formatted in /usr/share/doc.

If you still have a question or problem, please take the output of
`uname -a', along with any relevant error messages, and email it
as a question to the questions@FreeBSD.org mailing list. If you are
unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
manual page. If you are not familiar with manual pages, type `man man'.

You may also use sysinstall(8) to re-enter the installation and
configuration utility. Edit /etc/motd to change this login announcement.

nslug# ntpdate harmony
3 Aug 07:04:18 ntpdate[582]: step time server 10.0.0.6 offset 1217746845.716155 sec
slug# date
Sun Aug 3 07:04:19 UTC 2008
slug# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
10.0.0.8:/pe/slug 55871592 49741636 1660230 97% /
devfs 1 1 0 100% /dev
10.0.0.8:/pe 55871592 49741636 1660230 97% /pe

Minor mips reorg and plans for the future

In trying to bring the MIPS tree more into line with idealized FreeBSD practices, I've moved the SoC directories up one level in the hierarchy. Future changes will make it easier to define different boards, as well as stylizing these sorts of things across different CPU architectures. I need this for the Plat'Home OpenMicroServer work I've started on, so I'll use that to drive these changes.

For the curious, I've moved mips/mips32/adm5120 -> mips/adm5120, likewise with the idt, malta, and sentry5 directories. While all of these were, strictly speaking, mips32 CPUs, having them in their own subdirectory doesn't make a lot of sense. There's never going to be so many of them that having an extra layer of indirection is going to help. Also, mips32 is ambiguous. Does it mean the MIPS ISA known as MIPS32 and MIPS32r2? Does it mean that the ports are running in 32-bit mode? It wasn't clear where a 32-bit kernel for a mips64 processor would live. Also, the idt parts aren't completely mips32 compliant in a couple of minor details. Is that non-compliance enough to move them elsewhere?

The next set of changes will try to tease apart the boot loader environment from the SoC that it is running on. While many SoCs have their own bootloader, often times one may also run uboot on the same hardware. In addition, things like uboot are used in many different platforms, so duplicating that code will grow tiresome.

Once the boot loader stuff is better abstracted, the next set of problems will be how to create a system that can run on multiple different boards of the same SoC. At first, this will be statically compiled, but eventually it will be dynamic.

Another key component of this work, whose overall goal is to make it easier to add board support to FreeBSD embedded platforms, will be an overhaul of the hints system. While it can be made to work, there are some practical problems it solves poorly that are desirable to have solved better. For example, it has no good way to tie attributes or parameters to hardware. Today, one can only tie them to a driver instance, which itself can be tied to hardware.

Finally, the last planned component in this work is a simplification of creation of busses with attributes and fixed or semi-fixed children. You can't have variants of hints loaded today that say "If I'm running on a AT91RM9200, use this set, but if I'm runnning on an AT91SAM9620 use this other set of hints" to enumerate the busses. This makes it hard to achieve the dynamic selection of boot code to run needed to allow kernels to boot on a variety of boards.

Most of this is low-level code janitor stuff. However, it is code janitor stuff that's needed and will be important to FreeBSD's success in the future.

Adding a Serial Port to a D-LINK DIR-615

A while ago, I added a serial port to the D-LINK DIR-615 (HW Rev A1). Today, I'll document how I did it.

First, you'll need a level converter circuit. There are many on the network. You can find a good write up on them and links from adding a serial port to a NSLU2. I always use the Parallax USB2SER that I bought years ago. It is very reliable, and I've had several provide years of good service to me. They convert 3.3V directly into a USB serial port, which vastly simplifies power and cabling. Since that's what I used for this project, all the instructions are geared towards it.

First, you'll need to disassemble the D-LINK DIR-615. There's only two screws so this is easy. Place the D-Link with the label with the serial number face up. There are four rubber feet on the bottom of this unit. Carefully pry up the two nearest to the Ethernet ports. Remove the screws found under them. The screws are clearly visible in this photo.



Carefully remove the lid. It should pry off just above the plastic bar that runs over the ethernet ports. You'll see a green PC board. On the right hand side of the unit, you'll find a header with 4 pins labeled CON5. They are labeled VCC, TX, RX, GND. This is the serial port. In my unit, it was hidden under the internal antenna, so the photos show it moved over a bit. Note also in figure two the board says "DIR635B1" on it...



I connected these pins to a ribbon cable and ran them out to a header that my USB2SER could fit on. I connected them so that they would match up to my USB2SER which had pins in the order GND, RX, TX, RES. The last one is unconnected. TX should go to RX, RX to TX and GND to GND. I didn't connect VCC, since it wasn't needed. USB2SER gets its power from the USB port. I reconnected everything and plugged the USB2SER into my computer. I threaded the ribbon cable out through the holes in the plastic, which is what you see in the first photo. Here's everything hooked up. I didn't include a photo of the USB2SER plugged into the serial port of my laptop for obvious reasons...



I see that it has a uboot bootloader:

% tip ucom0
U-Boot 1.1.1 (Jan 19 2007 - 11:08:07)
CAMEO uBoot Linux Loader version: 1.3.0.0

DRAM CS[0] base 0x00000000   size  32MB
DRAM Total size  32MB
before entry mvFlashInit
Flash: flashStructGet manu 0x89 id 0x17
INTEL 28F640J3A (64 Mbit)
Size:  8 MB,Bus Width: 1, device Width: 1.
Flash base: 0xff800000,Number of Sectors: 64 Type: REGULAR.
[8192kB@ff800000] Flash:  8 MB
Addresses 20M - 0M are saved for the U-Boot usage.
Mem malloc Initialization (20M - 16M): Done
*** Warning - bad CRC, using default environment


Soc: 88F5181 B1
CPU: ARM926 (Rev 0) running @ 500Mhz
SysClock = 166Mhz , TClock = 166Mhz


USB 0: host mode
PCI 0: PCI Express Root Complex Interface
PCI 1: Conventional PCI, speed = 33000000
Net:   egiga0 [PRIME]
Hit any key to stop autoboot:  0
Marvell>>

So that's how you add a serial port.

D-LINK DIR-615

The other day I saw that Target was selling the D-LINK DIR-615 draft N router for $50.00 each. Since it had been a while since I purchased a wireless router to hack, I bought one. It turns out that this was a wise choice.

This router chip has an 88F5180NB1 CPU. This is a Marvell Orion SoC. Support for this family of processors is coming into the tree shortly.

There's a FS28F640 flash chip on board. This is a 64Mb/8MB of memory. With compression, FreeBSD can fit into this size. There appears to be 32MB of RAM, which is also enough to run FreeBSD.

There's a 88E6061-LAJ1 PHY chip which should be relatively easy to program.

Finally, it has a mini-pci slot with a 802.11n card based on the 88W8361P-BEM1 chip. Maybe a driver for this chip will come along. I've not investigated the GPL tarball to see if there's a driver in source for this, or just a binary module.

There's a GPL tarball available at the D-Link web site.

Finally, there's two interesting connectors. One looks to be a JTAG connector, the other looks to be a serial port.

My next step is to look into what's coming out of the serial port and what's in the GPL tarball for hints while I'm waiting for the Orion support to come into the tree.

FreeBSD/arm on a Linksys NSLU2 (slug)

Thanks to some encouragement from Gavin Atkinson, I have been able to boot the AVILA kernel on my Linksys NSLU2 (slug). I have it running of NFS root at 266MHz. Not super fast, but not too bad. I need to clean up a bunch of stuff still, but at least I have the basics working.

I'll post more in a few days, once I get a chance to clean up what I have and submit it to CVS. But this is very exciting...