Creating a quick DNS server with a Rapsberry Pi2 and FreeBSD 11.0-RC1

Raspberry Pi 2 and FreeBSD 11 and bind 9.9 for a name server

Just thought I'd outline the steps to bring a Raspberry Pi 2 up with FreeBSD and then create a DNS server with bind. This is for the Raspberry Pi2, but the only difference between this and any other supported board is the file you grab.

The TL;DR version

Download the image, use xz to decompress and dd to a SD card. Boot the SD card with a serial console, create a new account to log into. Install the appropriate packages. Configure named. Tell the world about it (router and registrar). Profit! All quite simple, though there were a few snags that are in the 'Long Version'

Long Version, so much detail

First, download the FreeBSD image from https://www.freebsd.org/where.html. I'm doing the SD Card Image for FreeBSD 11.0-RC1. This gave me the file FreeBSD-11.0-RC1-arm-armv6-RPI2.img.xz in my downloads area. If you're doing a different image, adjust below.

Next, I needed to create a microSD card for the RPi-2. I plugged a 4GB SD Card into my USB adapter, and the device appeared as da2 on my FreeBSD box.

 xz -d < ~/FreeBSD-11.0-RC1-arm-armv6-RPI2.img.xz | sudo dd of=/dev/da2 bs=1m

will uncompress it and splat it onto the SD card.

Next, I connected to my RPi2's serial port with tip and the Adafruit USB to TTY Adapter:

tip -115200 ucom1

When booting, I got this log. Yours might not be identical, but the login: at the end should be the same.

U-Boot 2015.04 (Aug 12 2016 - 16:50:57)

DRAM:  944 MiB

WARNING: Caches not enabled

RPI 2 Model B

MMC:   bcm2835_sdhci: 0

reading uboot.env

** Unable to read "uboot.env" from mmc0:1 **

Using default environment

In:    serial

Out:   lcd

Err:   lcd

Net:   Net Initialization Skipped

No ethernet found.

Hit any key to stop autoboot:  0 

Booting from: mmc 0 ubldr

reading ubldr

272013 bytes read in 213 ms (1.2 MiB/s)

## Starting application at 0x02000098 ...

Consoles: U-Boot console  

Compatible U-Boot API signature found @0x3ab4a4c8

FreeBSD/armv6 U-Boot loader, Revision 1.2

(root@releng2.nyi.freebsd.org, Fri Aug 12 17:05:59 UTC 2016)

DRAM: 944MB

Number of U-Boot devices: 1

U-Boot env: loaderdev='mmc 0'

Found U-Boot device: disk

  Checking unit=0 slice= partition=... good.

Booting from disk0s2a:

/boot/kernel/kernel data=0x72a864+0x17979c syms=[0x4+0x7e120+0x4+0x90fad]

Hit [Enter] to boot immediately, or any other key for command prompt.

Booting [/boot/kernel/kernel]...               

Using DTB provided by U-Boot at address 0x100.

Kernel entry at 0x0x2200100...

Kernel args: (null)

Copyright (c) 1992-2016 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 11.0-RC1 #0 r303979: Fri Aug 12 17:12:13 UTC 2016

    root@releng2.nyi.freebsd.org:/usr/obj/arm.armv6/usr/src/sys/RPI2 arm

FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)

VT: init without driver.

sema_sysinit

CPU: Cortex A7 rev 5 (Cortex-A core)

 Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext

 WB enabled LABT branch prediction disabled

LoUU:2 LoC:3 LoUIS:2 

Cache level 1: 

 32KB/64B 4-way data cache WB Read-Alloc Write-Alloc

 32KB/32B 2-way instruction cache Read-Alloc

Cache level 2: 

 512KB/64B 8-way unified cache WB Read-Alloc Write-Alloc

real memory  = 989851648 (943 MB)

avail memory = 956411904 (912 MB)

FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

random: entropy device external interface

kbd0 at kbdmux0

ofwbus0:

simplebus0: mem 0x3f000000-0x3fffffff on ofwbus0

local_intc0: mem 0x40000000-0x400000ff on simplebus0

generic_timer0: on ofwbus0

Timecounter "ARM MPCore Timecounter" frequency 19200000 Hz quality 1000

Event timer "ARM MPCore Eventtimer" frequency 19200000 Hz quality 1000

intc0: mem 0xb200-0xb3ff on simplebus0

bcmwd0: mem 0x10001c-0x100027 on simplebus0

gpio0: mem 0x200000-0x2000af on simplebus0

gpio0: read-only pins: 46,48-53.

gpio0: reserved pins: 48-53.

gpiobus0: on gpio0

gpioled0: at pin 35 on gpiobus0

gpioled1: at pin 47 on gpiobus0

gpioc0: on gpio0

iichb0: mem 0x205000-0x20501f on simplebus0

iicbus0: on iichb0

iic0: on iicbus0

iichb1: mem 0x804000-0x80401f on simplebus0

iicbus1: on iichb1

iic1: on iicbus1

spi0: mem 0x204000-0x20401f on simplebus0

spibus0: on spi0

bcm_dma0: mem 0x7000-0x7fff,0xe05000-0xe05fff on simplebus0

mbox0: mem 0xb880-0xb8bf on simplebus0

sdhci_bcm0: mem 0x300000-0x3000ff on simplebus0

mmc0: on sdhci_bcm0

uart0: mem 0x201000-0x201fff on simplebus0

uart0: console (115200,n,8,1)

vchiq0: mem 0xb800-0xb84f on simplebus0

vchiq: local ver 8 (min 3), remote ver 8.

pcm0: on vchiq0

bcm283x_dwcotg0: mem 0x980000-0x99ffff on simplebus0

usbus0 on bcm283x_dwcotg0

cpulist0: on ofwbus0

cpu0: on cpulist0

bcm2835_cpufreq0: on cpu0

cpu1: on cpulist0

cpu2: on cpulist0

cpu3: on cpulist0

fb0: on ofwbus0

fbd0 on fb0

VT: initialize with new VT driver "fb".

fb0: 1824x984(1824x984@0,0) 24bpp

fb0: fbswap: 1, pitch 5472, base 0x3d359000, screen_size 5428224

cryptosoft0:

Timecounters tick every 10.000 msec

usbus0: 480Mbps High Speed USB v2.0

ugen0.1: at usbus0

uhub0: on usbus0

mmcsd0: 4GB at mmc0 41.6MHz/4bit/65535-block

bcm2835_cpufreq0: ARM 600MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF

Release APs

Trying to mount root from ufs:/dev/ufs/rootfs [rw]...

warning: no time-of-day clock registered, system time will not be set accurately

uhub0: 1 port with 1 removable, self powered

Growing root partition to fill device

GEOM_PART: mmcsd0s2 was automatically resized.

  Use `gpart commit mmcsd0s2` to save changes or `gpart undo mmcsd0s2` to revert them.

mmcsd0s2 resized

ugen0.2: at usbus0

uhub1: on usbus0

uhub1: MTT enabled

mmcsd0s2a resized

super-block backups (for fsck_ffs -b #) at:

 1994944, 2493632, 2992320, 3491008,uhub1: 5 ports with 4 removable, self powered

 3989696, 4488384, 4987072, 5485760,

 5984448, 6483136, 6981824, 7480512

ugen0.3: at usbus0

smsc0: on usbus0

smsc0: chip 0xec00, rev. 0002

miibus0: on smsc0

ukphy0: PHY 1 on miibus0

ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto

ue0: on smsc0

ue0: Ethernet address: b8:27:eb:xx:xx:xx

ugen0.4: at usbus0

ukbd0: on usbus0

kbd1 at ukbd0

/etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one

Setting hostuuid: 3b610f45-60b0-11e6-85fc-b827eb30ec4b.

Setting hostid: 0xd150b3f7.

Starting file system checks:

/dev/ufs/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS

/dev/ufs/rootfs: clean, 745130 free (266 frags, 93108 blocks, 0.0% fragmentation)

Mounting local filesystems:.

ELF ldconfig path: /lib /usr/lib /usr/lib/compat

random: unblocking device.

Soft Float compatibility ldconfig path:

Setting hostname: rpi2.

Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED

Feeding entropy: .

smsc0: chip 0xec00, rev. 0002

ue0: link state changed to DOWN

ue0: link state changed to UP

Starting Network: lo0 ue0.

lo0: flags=8049 metric 0 mtu 16384

options=600003

inet6 ::1 prefixlen 128 

inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 

inet 127.0.0.1 netmask 0xff000000 

groups: lo 

nd6 options=21

ue0: flags=8843 metric 0 mtu 1500

options=80009

ether b8:27:eb:30:ec:4b

media: Ethernet autoselect (100baseTX )

status: active

nd6 options=29

Starting devd.

Starting dhclient.

DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 3

DHCPOFFER from 10.0.0.5

DHCPOFFER from 10.0.0.3

DHCPREQUEST on ue0 to 255.255.255.255 port 67

DHCPACK from 10.0.0.5

bound to 10.0.0.245 -- renewal in 3600 seconds.

add host 127.0.0.1: gateway lo0 fib 0: route already in table

add host ::1: gateway lo0 fib 0: route already in table

add net fe80::: gateway ::1

add net ff02::: gateway ::1

add net ::ffff:0.0.0.0: gateway ::1

add net ::0.0.0.0: gateway ::1

Generating host.conf.

Creating and/or trimming log files.

Starting syslogd.

Clearing /tmp (X related).

Updating motd:.

Mounting late filesystems:.

Configuring vt: blanktime.

Generating RSA host key.

2048 SHA256:FXp0GIg9YQO8QKDV4laoIrXLhUPcBQPaYeusHOnn4wI root@rpi2 (RSA)

Generating ECDSA host key.

256 SHA256:lYyCJFYZHx2W8slGyhe8CF6GIc8ejZZWcO4wTlRf/YI root@rpi2 (ECDSA)

Generating ED25519 host key.

256 SHA256:L2wqTBkNdGahTGswcpzG4ysvhU9uCPyjBtLGLczi6fI root@rpi2 (ED25519)

Performing sanity check on sshd configuration.

Starting sshd.

Starting cron.

Starting background file system checks in 60 seconds.

mount: /dev/ufs/rootfs: Device busy

Fri Aug 12 17:15:15 UTC 2

FreeBSD/arm (rpi2) (ttyu0)

login: root

Password:

Aug 12 17:15:19 rpi2 login: ROOT LOGIN (root) ON ttyu0

FreeBSD 11.0-RC1 (RPI2) #0 r303979: Fri Aug 12 17:12:13 UTC 2016

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/

Security Advisories:   https://www.FreeBSD.org/security/

FreeBSD Handbook:      https://www.FreeBSD.org/handbook/

FreeBSD FAQ:           https://www.FreeBSD.org/faq/

Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/

FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/

directory, or can be installed later with:  pkg install en-freebsd-doc

For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a

Please include that output and any error messages when posting questions.

Introduction to manual pages:  man man

FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.

root@rpi2:~ # adduser

Username: imp

Full name: M Warner Losh

Uid (Leave empty for default): 1234

Login group [imp]: 

Login group is imp. Invite imp into other groups? []: wheel

Login class [default]: 

Shell (sh csh tcsh nologin) [sh]: tcsh

Home directory [/home/imp]: 

Home directory permissions (Leave empty for default): 

Use password-based authentication? [yes]: 

Use an empty password? (yes/no) [no]: 

Use a random password? (yes/no) [no]: 

Enter password: 

Enter password again: 

Lock out the account after creation? [no]: 

Username   : imp

Password   : *****

Full Name  : M Warner Losh

Uid        : 1234

Class      : 

Groups     : imp wheel

Home       : /home/imp

Home Mode  : 

Shell      : /bin/tcsh

Locked     : no

OK? (yes/no): yes

adduser: INFO: Successfully added (imp) to the user database.

Add another user? (yes/no): no

Goodbye!

root@rpi2:~ # logout

FreeBSD/arm (rpi2) (ttyu0)

login: 

 So, now we have a remote login (or we could have used the default freebsd/freebsd user).

Next, we need to bootstrap the packages. In the RC1 release there's a snag. The quarterly release hasn't been created yet, so you have to override the location:

# env PACKAGESITE=pkg+http://pkg.FreeBSD.org/FreeBSD:11:armv6/latest pkg boostrap

The package management tool is not yet installed on your system.Do you want to fetch and install it now? [y/N]: yBootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:armv6/latest, please wait...Verifying signature with trusted certificate pkg.freebsd.org.2013102301... doneInstalling pkg-1.8.5_1...Extracting pkg-1.8.5_1: 100%

root@rpi2:/home/imp # 

Now, in the RC1 release, there's another snag, you have to change quarterly to latest in /etc/pkg/FreeBSD.conf. Once you do that, you can continue. Since this is a simple box, I'm just installing sudo, bind99 and emacs-nox11 on it. Don't need anything more than that.

root@rpi2:/home/imp # pkg install sudo bind99 emacs-nox11

Updating FreeBSD repository catalogue...Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    Fetching packagesite.txz: 100%    5 MiB 525.0kB/s    00:09    Processing entries: 100%FreeBSD repository update completed. 21349 packages processed.The following 18 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:sudo: 1.8.17bind99: 9.9.9P1emacs-nox11: 24.5_3,3gettext-runtime: 0.19.8.1indexinfo: 0.2.4libxml2: 2.9.3libedit: 3.1.20150325_2,1idnkit: 1.0_5gnutls: 3.4.13nettle: 3.2gmp: 5.1.3_3ca_root_nss: 3.22.2libtasn1: 4.8trousers: 0.3.13tpm-emulator: 0.7.4_1p11-kit: 0.23.2libffi: 3.2.1libidn: 1.31
The process will require 179 MiB more space.48 MiB to be downloaded.
Proceed with this action? [y/N]: yFetching sudo-1.8.17.txz: 100%  835 KiB 427.4kB/s    00:02    Fetching bind99-9.9.9P1.txz: 100%    4 MiB 583.4kB/s    00:08    Fetching emacs-nox11-24.5_3,3.txz: 100%   36 MiB 574.9kB/s    01:06    Fetching gettext-runtime-0.19.8.1.txz: 100%  142 KiB 145.5kB/s    00:01    Fetching indexinfo-0.2.4.txz: 100%    5 KiB   5.0kB/s    00:01    Fetching libxml2-2.9.3.txz: 100%  735 KiB 376.5kB/s    00:02    Fetching libedit-3.1.20150325_2,1.txz: 100%  107 KiB 110.1kB/s    00:01    Fetching idnkit-1.0_5.txz: 100%  182 KiB 185.9kB/s    00:01    Fetching gnutls-3.4.13.txz: 100%    2 MiB 501.4kB/s    00:04    Fetching nettle-3.2.txz: 100%    1 MiB 378.8kB/s    00:03    Fetching gmp-5.1.3_3.txz: 100%  390 KiB 399.5kB/s    00:01    Fetching ca_root_nss-3.22.2.txz: 100%  323 KiB 330.3kB/s    00:01    Fetching libtasn1-4.8.txz: 100%  584 KiB 298.9kB/s    00:02    Fetching trousers-0.3.13.txz: 100%  390 KiB 399.6kB/s    00:01    Fetching tpm-emulator-0.7.4_1.txz: 100%   18 KiB  18.0kB/s    00:01    Fetching p11-kit-0.23.2.txz: 100%  199 KiB 203.4kB/s    00:01    Fetching libffi-3.2.1.txz: 100%   33 KiB  33.4kB/s    00:01    Fetching libidn-1.31.txz: 100%  196 KiB 200.9kB/s    00:01    Checking integrity... done (0 conflicting)[1/18] Installing indexinfo-0.2.4...[1/18] Extracting indexinfo-0.2.4: 100%[2/18] Installing gmp-5.1.3_3...[2/18] Extracting gmp-5.1.3_3: 100%[3/18] Installing gettext-runtime-0.19.8.1...[3/18] Extracting gettext-runtime-0.19.8.1: 100%[4/18] Installing ca_root_nss-3.22.2...[4/18] Extracting ca_root_nss-3.22.2: 100%[5/18] Installing libtasn1-4.8...[5/18] Extracting libtasn1-4.8: 100%[6/18] Installing tpm-emulator-0.7.4_1...===> Creating groups.Creating group '_tss' with gid '601'.===> Creating usersCreating user '_tss' with uid '601'.[6/18] Extracting tpm-emulator-0.7.4_1: 100%[7/18] Installing libffi-3.2.1...[7/18] Extracting libffi-3.2.1: 100%[8/18] Installing nettle-3.2...[8/18] Extracting nettle-3.2: 100%[9/18] Installing trousers-0.3.13...===> Creating groups.Using existing group '_tss'.===> Creating usersUsing existing user '_tss'.[9/18] Extracting trousers-0.3.13: 100%[10/18] Installing p11-kit-0.23.2...[10/18] Extracting p11-kit-0.23.2: 100%[11/18] Installing libidn-1.31...[11/18] Extracting libidn-1.31: 100%[12/18] Installing libxml2-2.9.3...[12/18] Extracting libxml2-2.9.3: 100%[13/18] Installing libedit-3.1.20150325_2,1...[13/18] Extracting libedit-3.1.20150325_2,1: 100%[14/18] Installing idnkit-1.0_5...[14/18] Extracting idnkit-1.0_5: 100%[15/18] Installing gnutls-3.4.13...[15/18] Extracting gnutls-3.4.13: 100%[16/18] Installing sudo-1.8.17...[16/18] Extracting sudo-1.8.17: 100%[17/18] Installing bind99-9.9.9P1...[17/18] Extracting bind99-9.9.9P1: 100%[18/18] Installing emacs-nox11-24.5_3,3...[18/18] Extracting emacs-nox11-24.5_3,3: 100%Message from ca_root_nss-3.22.2:********************************* WARNING *********************************
FreeBSD does not, and can not warrant that the certification authoritieswhose certificates are included in this package have in any way beenaudited for trustworthiness or RFC 3647 compliance.
Assessment and verification of trust is the complete responsibility of thesystem administrator.
*********************************** NOTE **********************************
This package installs symlinks to support root certificates discovery bydefault for software that uses OpenSSL.
This enables SSL Certificate Verification by client software without manualintervention.
If you prefer to do this manually, replace the following symlinks witheither an empty file or your site-local certificate bundle.
  * /etc/ssl/cert.pem  * /usr/local/etc/ssl/cert.pem  * /usr/local/openssl/cert.pem
***************************************************************************Message from trousers-0.3.13:To run tcsd automatically, add the following line to /etc/rc.conf:
tcsd_enable="YES"
You might want to edit /usr/local/etc/tcsd.conf to reflect your setup.
If you want to use tcsd with software TPM emulator, use the followingconfiguration in /etc/rc.conf:
tcsd_enable="YES"tcsd_mode="emulator"tpmd_enable="YES"
To use TPM, add your_account to '_tss' group like following:
# pw groupadd _tss -m your_accountMessage from idnkit-1.0_5:===>   NOTICE:
The idnkit port currently does not have a maintainer. As a result, it ismore likely to have unresolved issues, not be up-to-date, or even be removed inthe future. To volunteer to maintain this port, please create an issue at:
https://bugs.freebsd.org/bugzilla
More information about port maintainership is available at:
https://www.freebsd.org/doc/en/articles/contributing/ports-contributing.html#maintain-portMessage from bind99-9.9.9P1:***********************************************************************            _  _____ _____ _____ _   _ _____ ___ ___  _   _         **           / \|_   _|_   _| ____| \ | |_   _|_ _/ _ \| \ | |        **          / _ \ | |   | | |  _| |  \| | | |  | | | | |  \| |        **         / ___ \| |   | | | |___| |\  | | |  | | |_| | |\  |        **        /_/   \_\_|   |_| |_____|_| \_| |_| |___\___/|_| \_|        **                                                                    **   BIND requires configuration of rndc, including a "secret" key.   **    The easiest, and most secure way to configure rndc is to run    **   'rndc-confgen -a' to generate the proper conf file, with a new   **            random key, and appropriate file permissions.           **                                                                    **     The /usr/local/etc/rc.d/named script will do that for you.     **                                                                    ***********************************************************************root@rpi2:/home/imp # 

Although it says that the rc.d named script will do it, I went ahead and did the rndc-confgen -a:

root@rpi2:/home/imp # rndc-confgen -awrote key file "/usr/local/etc/namedb/rndc.key" 

next, we need to copy over the master zone and setup named.conf. Briefly, I commented out the part that binds to localhost and added a pointer to the master file for my zone. I setup stuff so secondaries could transfer out, and I could cross-mirror the secondaries.

So, once it was all running right, I changed rc.conf to enable it (as well as tweak the hostname from the default):

named_enable=yes

Now the fun part: telling my router to forward DNS packets (port 53) to my FreeBSD box, and then telling my registrar the new IP address of my DNS server. And after the obligatory password snafu, I was able to get everything up and happy.

Removed the default 'freebsd' account and changed the root password to something useful, and I was done.

A final though

I used to think I'd need to pay someone to host DNS for me. And that used to be cheaper in someone else's colo. However, with dirt cheap boards like the Raspberry Pi 2 and others like it, now I can host it myself for not much more than paying someone else do do it. And setting up the server was super easy. Wish blogspot would allow me to more easily attach text files, but it's an imperfect world.

Creating Logos for u-boot images

The Default Logo: Tux

By default, Das U-boot uses a penguin logo when it first starts up. This default looks something like this

While it's cool to have this for Linux users, FreeBSD users would like to see something more relevant. I recently got a pcDuino 3 board that I wanted to get FreeBSD running on. When I saw Tux still greeting me, I knew I had to do something. If nothing else, we should be branding the images made by the project with project logos and not the default which advertises a competitor.

The Quest

So, I set off on a quest to find the logo that would be best. It turns out to be not as simple a question as I thought jumping into this as I thought.

Logo vs Mascot

Traditionally, FreeBSD has had both the logo and the mascot as part of its boot sequence. The FreeBSD daemon that was used was based on artwork from the Walnut Creek CDROM which in turn was based on artwork drawn for Kirk McKusick by Lassiter. I realized that I'd have to do both. I'd make the logo the default, but also make it possible to have the mascot. While the Lassiter image appears to be the official mascot of the project, the Walnut Creek CDROM image is more widely viewed as the mascot of the FreeBSD project, so I had to select which one to use, or add yet another option.

Finding Good Artwork

The needs of u-boot is simple. Generally, you have an 80x80 image that's splashed up on the screen. While there is some wiggle room for the size of the icon, irregular sized icons just look sloppy in comparison to the default (there's several in the u-boot distribution to choose from, so I tried each one and was left dissatisfied each time).

A few web searches later, I found the best three examples to work from that I could: one with the FreeBSD logo, one with the Lassiter image, and a decent one of the Walnut Creek CDROM cd inserts and various books on FreeBSD (the older image from the 2.0 CDROM, not the newer one with daemon walking out of the CD that was used after).

The Lassiter image was by far the worst of the lot, but it was the best I could find. It proved too difficult to work with, so I eliminated it from my work. The logo and the Walnut Creek daemons would have to do.

Creating BMP images

Turns out this is harder than I would have thought.

To do the FreeBSD-associated daemon, I grabbed a copy from wikipedia's BSD daemon article and did the following:

convert freebsd-daemon.jpg -bordercolor white -channel RGBA -fuzz 20% -fill black -floodfill +0+0 white -resize 80x80\! -colors 200 -alpha off -type palette -compress none freebsd-daemon-1.bmp

To do the FreeBSD logo, I grabbed a copy from the FreeBSD web site and did the following:

convert logo-full.png -crop 172x172+2+2\! -resize 80x80 -colors 200 -alpha off -type palette -compress none -transparent-color black logo-full.bmp

This astute observer will notice a few things here. First, both images were 80x80. That's the default size. The u-boot tool, even with my fixes, only works with widths that are an even multiple of 4. I did 200 colors, but the limit is 240 (256 - 16).  Due to the way that uboot encodes the images, more than around 100 colors isn't going to have a big impact for either of these images (it takes the upper nibble from R G and B value that, and only that, so nuanced differences in colors don't matter). There's likely some fancy mode to optimize color generation in ImageMagick (where the convert program comes from), but I stopped before finding it.

Fixing U-Boot to have it join the 21st century

Turns out the bmp_logo utility that comes with u-boot has some serious limitations. First, it won't detect when you have a width that isn't a multiple of 4, and will generate a bad file instead that you won't discover is bad until you boot the box. Next, it doesn't check to see if the bmp is a compressed one or not with the same result. Ditto for number of colors. Same result. Those are all annoying, but the worst thing of all is that it won't handle any BMP that's newer than Windows 95. The BMP format has a field in a header that tells it how big the image hader is. bmp_logo, as distributed, assumes this length (40), but that only works for win95 and older (but not too old, super-antient BMPs likely would fail too). Any modern tool will generate the newer format (like ImageMagick does).

I've fixed the bmp_logo program locally, and merged a copy of the diffs into FreeBSD's repo for the sysutils/u-boot-olimex-a20-som-evb port (so all a20 and newer allwinner ports). I'm working to get this upstreamed.

The Results

Logo

Here's a slightly blurry shot of the logo I settled on. https://people.freebsd.org/~imp/20160814_104146.jpg

[update]
http://people.freebsd.org/~imp/20160815_234153.jpg uses the newest FreeBSD Foundation recommended logo...

The Daemon Formerly Known as Chuckie

Here's a blurry shot of the Walnut Creek daemon.  https://people.freebsd.org/~imp/20160814_124703.jpg

u-boot changes

I've created patches for the A20 Allwinnner ports. They will be committed shortly. I'll also be working to get these upstream into u-boot.

FreeBSD exposure on recent glibc DNS overflow

Recently, there's been much talk in the news about CVE-2015-7547 affecting almost all Linux-based routers and many distributions.

A quick analysis shows that FreeBSD's native libraries are completely unaffected. If you are running Linux emulation binaries, you may be at risk, and should update the linux emulation package.

Dag-Erling Smørgrav has a good blog with all the details.

Details on coming automatic module loading in FreeBSD

Automatic Module Loading

For a long time, I've wanted to add better, automatic module loading to FreeBSD. This past year, I started implementing that feature. Time and work pressures prevented me from completing it.

Some background

Every device that we have in our systems is enumerated in one of two ways. Either the bus gives us a list of all the devices, with certain per-device attributes (called plug and play data), or the device is attached through some other means. These latter devices are beyond the scope of this work, and there's generally very few of them in the system, and they aren't optional. The other devices, sometimes called 'self-enumerating' devices, have enough plug and play data for drivers in the system to decide if that driver can drive them or not. Most operating systems assign devices to drivers using this data. Some have the data encoded into tables in the filesystem (Windows, OS  X and Solaris), while others encode the data into the drivers (FreeBSD, NetBSD, OpenBSD, Linux, Dragon Fly BSD), though Linux offers some hybridization when it comes to certain devices. Except for FreeBSD, all these other OSes are beyond the scope of this work. And there's certainly room for debate over which approach is best, but we'll leave that behind as well.

Some busses on FreeBSD, like USB and PC Card, have very stylized probe routines. Drivers for devices on these busses generally call some bus-provided routine to match the device against a table that's basically the same for all drivers (though there's usually some stylized way to attach extra data). These drivers are easy to adapt to this new scheme because while some custom code needs to be written for each bus, each driver of that bus can generally use a macro to implement marking the PNP data (more on what marking means in a bit).

Other busses, like PCI, leave it entirely to the driver. So most of the drivers in the system have written their own matching routines that essentially loop through a table matching some attribute the bus provides to decide if the device is for them. These busses are harder to adapt.

But either way, most all drivers have some table of plug and play data that they use to filter their choice of devices. Since drivers are compiled into modules, this means most modules on the system have this data too. However, given the chaotic nature of the different busses, it's impossible for a program to find this data, unless it has very specific knowledge of each and every driver on the system (though for some classes of drivers, like USB and PC Card, it needn't know everything).

Finally, switching gears a bit, we have modules. Modules in the system record dependencies on other modules in the system using special macros. When the modules are installed, kldxref(8) runs through all of them, extracting these dependencies into a file called linker.hints that lives in the directory kernel and modules reside in.

General Theory

So, with that background, it's time to explore the design. What if we could take that chaotic state of affairs and somehow tame it. If we could create macros that we could use to mark the plug and play data, as well as associate various binary bits with the plug and play attribute provided by the bus driver? What if we could create records in the modules similar to what is used to mark module dependencies? Then, kldxref(8) would be able to comb through this data, record it in linker.hints and we'd need little other modifications to the system to make this data readily accessible.

That's exactly what my changes do. New types of records are inserted into the special section of modules that describe the plug and play table in the driver. These records contain a header that contains the length of each table entry, a pointer to the first entry, and a tiny little "script" or "description" of the table that ties this binary data to the bus-provided plug and play data. Each bus in the system that is of the stylized type described above defines its own macros to help its client drivers mark the data. Since all the data is the same for all the drivers, this means client drivers don't need to reinvent the wheel. Since we also pass the length of each table entry, drivers can use the common pattern of having the common data first, followed by whatever other data it needs for each device in the table.

However, that leaves more work for the non-stylized busses. While a few of these drivers have been converted, many remain.

Details about the marking

The first place to look for the details is . The first user-visible bit is a macro MODULE_PNP_INFO

#define MODULE_PNP_INFO(d, b, unique, t, l, n)

"d" is the description of the table (more on that below). "b" is the name of the bus. Unique is a unique string (typically the driver name). "t" is a pointer to the plug and play table. "l" is the length of each entry in the table. "n" is the number of entries.

The description is of the general form  (TYPE:pnp_name[/pnp_name];)* where TYPE is one of the following:

• U8      uint8_t element
• V8      like U8 and 0xff means match an
• G16    uint16_t element, any value >= matches
• L16     uint16_t element, any value <= matches
• M16    uint16_t element, mask of which of the following fields to use.
• U16     uint16_t element
• V16     like U16 and 0xffff means match any
• U32     uint32_t element
• V32     like U32 and 0xffffffff means match any
• W32     Two 16-bit values with first pnp_name in LSW and second in MSW
• Z       pointer to a string to match exactly
• D       like Z, but is the string passed to device_set_descr()
• P       A pointer that should be ignored
• E       EISA PNP Identifier (in binary, but bus publishes string)
• K       Key for whole table. pnp_name=value. must be last, if present.

The pnp_name "#" is reserved for other fields that should be ignored.

All this is a bit complicated. However, it can be simplified greatly for the buses that are stylized. For example PC Card, the PCCARD_PNP_INFO macro just takes a pointer to the first element and figures the rest out from there (you can see how in sys/dev/pccard/pccardvar.h).

kldxref(8)

Kldxref(8) has been modified to find all these entries. The above description is fairly complex, but covers all known tables in the current system. kldxref takes the above, and filters it into a much smaller subset by expanding different fields in a format more suited to quick parsing. It parses into the following format:

The format output is the simplified string from this routine in the same basic format as the pnp string, as documented in sys/module.h (and above). First a string describing the format is output, the a count of the number of records, then each record. The format string also describes the length of each entry (though it isn't a fixed length when strings are present).

type    Output          Meaning

   I       uint32_t        Integer equality comparison

   J       uint32_t        Pair of uint16_t fields converted to native byte order. The two fields both must match.

  G       uint32_t        Greater than or equal to

   L       uint32_t        Less than or equal to

  M       uint32_t        Mask of which fields to test. Fields that take up space increment the count. This field must be first, and resets the count.

   D       string          Description of the device this pnp info is for

   Z       string          pnp string must match this

   T       nothing         T fields set pnp values that must be true for the entire table.

Values are packed the same way that other values are packed in this file. Strings and int32_t's start on a 32-bit boundary and are padded with 0 bytes. Objects that are smaller than uint32_t are converted, without sign extension to uint32_t to simplify parsing downstream.

Current State

While the recording side is fairly well finished and committed to the tree, the rest of it is still up in the air. This section describes suggested future work for anybody wishing to help.

The easiest thing to do is to convert a few drivers to record this info. This conversion usually goes fairly quickly after you've found a similar driver that's been converted. Some drivers "save" space by matching the vendor code, for example, in code, while the device is matched from a table. When converting these drivers, you need to add the vendor code to each line in the table, and modify the code to get the vendor from the table.

Some buses have few enough drivers that it would be beneficial to adapt them to a stylized bus. simplebus being an obvious candidate. While most of the SoCs that have simplebus use a monolithic kernel, it's never too early to plan for a more generic and modular future. Some work in this area is in review.

There's three different strategies to use this data. First, the boot loader already reads in the linker.hints file. It could be modified to parse this data (it currently ignores it) and look at the PCI devices in the system. This leaves a number of holes, however, and loading drivers from the bootloader currently has significant performance issues. Second, the kernel could parse this file and load drivers as needed. However, this is far from straight forward in the kernel, since module loading needs to be queued until after / is available, and even after boot, some insertion events may happen in contexts that won't allow for modules to be loaded directly. Third, a user land program (perhaps devd(8)?) could parse the loader.hints file and create devd.conf scripts. USB currently has a program that will generate its hints based on ELF sections, a design which informed the current implementation. Its generator knows the format, which the current design hopes to avoid.

So a good start has been made, but more work is needed before we can ship only a MINIMAL kernel with modules. See this space for more info in the future.

Good, cheap scope.

I recently found a good scope for my hacking needs on Banggood. Thought you might be interested in it. You can find it here. It's the All-sun EM125 25MHz 2 in1 Mini Handheld Digital Oscilloscope + Multimeter.

TL;DR: Great value for $100! I love it for portable use.

I've used this now for a few weeks. My needs for a scope are modest. I need to see what wave forms are going on in different circuits and/or trying to figure out baud rates for stubborn serial lines. This scope gives me that and more. It also has all the features of a high-end multi-meter, including capacitance. And has a DMM mode that is easy to use.

The scope is solidly constructed. Normally at the low end you have to put up with funky adapters, fragile connection, light-wight (flimsy) construction. The construction of this device suffers from none of those defects. It is as solid as any of the higher end test gear I've used over the years. The fit and finish is quite nice. The aesthetics are pleasing without being obnoxious. Again, a pleasant surprise given the cost. The bag that comes with it is durable and holds all the accessories that come with the scope with some room to spare for a few extras.

While it lacks some of the features found in higher end scopes, and I would never think to make high precision measurements with it to calibrate systems, it gives me enough of a window into what's going on that it is well worth the money. Years ago I had a low-end Tektronics 5MHz scope that set me back more than $100 and wasn't half as easy to use as this. The only feature I miss from it with this one is the second channel for triggering / comparison.

It doesn't have a fancy color display like some of the cheaper scopes I looked at. However, it has a higher sampling bandwidth. The display is crisp and useful. The actual performance of the scope is better than the color ones costing twice as much.

The only down side for a native English speaker is that some of the menus on it are in Chinese. And the manual is in Chinese. The scope is simple enough to use that a little trial and error was all that was needed to get the basics. And I used my smart phone and google translate to work out some of the other bits.

For about US$100, I think it's an excellent value. If you are looking for a cheap scope that's portable and gives you good results, this one is for you. Don't expect the precision of a $1k or $10k scope, though. It lacks a second channel, calibration inputs and the display is also a bit small for high precision work. It's worked out great for trouble shooting and reverse engineering.

Hard Float API coming soon by default to armv6

All the CPUs that FreeBSD supports have hard floating point in them. We've supported hard float for quite some time in the FreeBSD kernel. However, by default, we still use a soft-float ABI. All the binaries use hardware floating point to do the math, but the calling conventions between routines passes the floating point values in integer registers to remain compatible with the soft-float libraries of the past. A new armv6hf was created, but that caused some issues with some ports, and the meaning of 'soft float' sadly was ambiguous between the soft-float ABI, and the soft-float libraries that implement floating point when there's no hardware FPU.

Over the spring and summer, I fixed ld.so so that it  can load both soft ABI and hard ABI libraries on the same system, depending markings in the binaries themselves. Soft float ABI and hard float ABI binaries have different flags in the ELF headers, so it is relatively straight forward to know which is which. Over the summer, I committed changes to the kernel to pass the relevant flags from the header to user land (since for a variety of technical reasons, once you get to user land, it's impossible to open the binary you are executing in the general case). I had planned on committing the ld.so bits as well, but my job got busy and now it's winter.

So, in the coming days, I'll commit the first set of changes to move to armv6 as a hard float ABI by default. The kernel doesn't care: it can execute both. The new ld.so will allow you to transition through this change by allowing old, compat soft ABI libraries to co-exist on the system with new hard ABI libraries. This change alone isn't enough, but it will be good to get it out into circulation.

After that, a number of changes will follow. Similar to the LIB32 stuff for x86, mips and powerpc, there will be a LIBSOFT build that you can enable on armv6. This will create the libraries needed for the transition, and will be used if we need to generate compat binaries for this stuff (the hope is that the number of old binaries that need to run on the new system will be small enough that it can be treated as a special-needs case, not one that's needed by default). The normal armv6 build will switch to hard float ABI by default. armv6hf will remain in the build as a compatibility arch, and will be removed before FreeBSD 11 since it was never in FreeBSD 10 (and could never be in FreeBSD 10 due to compiler choices). TARGET_CPUTYPE will grow a new type or two for those folks with really special needs that need to generate wither soft-float libraries, or soft-float ABI libraries (and this will be the mechanism used to support LIBSOFT builds).

NanoBSD reved up for embedded

NanoBSD Enhancements

I've been experimenting with NanoBSD enhancements for embedded. Eventually, they will be in the mainline of NanoBSD, but for now I've segregated them in a separate embedded subdir. The goal is to drive creation of build-agnostic tools to help build images for the different systems that FreeBSD supports. I have a mix of embedded boards and qemu configurations. Many don't work, but I hit a milestone tonight. I was able to boot an image on my RPi built without privs using NanoBSD.

To try it out, you'll need to grab FreeBSD-current and from the top of the tree do

cd tools/tools/nanobsd/embedded

sh ../nanobsd.sh -c rpi.cfg

This will create a directory at the same level as your top level FreeBSD source called rpi-b. Under it will be obj/_.disk.image.rpi-b. You can dd this onto a SD card (any size) and boot it on your RPi-B. It will resize the last partition to span the rest of the disk on boot and be a generic system. There's still a few problems, and I'm still working out how to get packages onto the system at build time, but its useful enough that people may want to play with it. Enjoy.

P.S. Here's a dmesg from the boot:

U-Boot 2015.10 (Nov 09 2015 - 02:54:46 +0000)

DRAM:  480 MiB
RPI Model B rev2
MMC:   bcm2835_sdhci: 0
reading uboot.env

** Unable to read "uboot.env" from mmc0:1 **
Using default environment

In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
reading uEnv.txt
0 bytes read in 9 ms (0 Bytes/s)
Hit any key to stop autoboot:  0 
starting USB...
USB0:   Core Release: 2.80a
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
       scanning usb for ethernet devices... 1 Ethernet Device(s) found
Booting from: mmc 0 ubldr.bin
reading ubldr.bin
214684 bytes read in 36 ms (5.7 MiB/s)
## No elf image at address 0x00200000
## Starting application at 0x00200000 ...
Consoles: U-Boot console  
Compatible U-Boot API signature found @1db474d0

FreeBSD/armv6 U-Boot loader, Revision 1.2
(imp@zooty, Wed Dec 23 17:23:10 MST 2015)

DRAM: 480MB
Number of U-Boot devices: 2
U-Boot env: loaderdev='mmc 0'
Found U-Boot device: disk
  Checking unit=0 slice= partition=... good.
Booting from disk0s3a:
/boot/kernel/kernel text=0x587d2c data=0x54f24+0xe639c syms=[0x4+0xc6460+0x4+0x94335]

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by U-Boot at address 0x100.
Kernel entry at 0x400180...
Kernel args: (null)
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r292645 0ff2d15e64f9+ qbase/qtip/rpi/tip: Wed Dec 23 17:29:02 MST 2015
    imp@zooty:/usr/home/imp/FreeBSD/rpi-b/obj/arm.armv6/usr/home/imp/FreeBSD/head/sys/RPI-B arm
FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906
VT: init without driver.
sema_sysinit
CPU: ARM1176JZ-S rev 7 (ARM11J core)
 Supported features: ARM_ISA THUMB2 JAZELLE ARMv4 Security_Ext
 WB enabled LABT branch prediction enabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 503312384 (479 MB)
avail memory = 481800192 (459 MB)
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: 
simplebus0:  mem 0x20000000-0x20ffffff on ofwbus0
cpulist0:  on ofwbus0
cpu0:  on cpulist0
bcm2835_cpufreq0:  on cpu0
intc0:  mem 0xb200-0xb3ff on simplebus0
systimer0:  mem 0x3000-0x3fff irq 8,9,10,11 on simplebus0
Event timer "BCM2835-3" frequency 1000000 Hz quality 1000
Timecounter "BCM2835-3" frequency 1000000 Hz quality 1000
bcmwd0:  mem 0x10001c-0x100027 on simplebus0
gpio0:  mem 0x200000-0x2000af irq 57,59,58,60 on simplebus0
gpio0: read-only pins: 46-53.
gpio0: reserved pins: 48-53.
gpiobus0:  on gpio0
gpioled0:  at pin 16 on gpiobus0
gpioc0:  on gpio0
iichb0:  mem 0x205000-0x20501f irq 61 on simplebus0
iicbus0:  on iichb0
iic0:  on iicbus0
iichb1:  mem 0x804000-0x80401f irq 61 on simplebus0
iicbus1:  on iichb1
iic1:  on iicbus1
spi0:  mem 0x204000-0x20401f irq 62 on simplebus0
spibus0:  on spi0
bcm_dma0:  mem 0x7000-0x7fff,0xe05000-0xe05fff irq 24,25,26,27,28,29,30,31,32,33,34,35,36 on simplebus0
mbox0:  mem 0xb880-0xb8bf irq 1 on simplebus0
sdhci_bcm0:  mem 0x300000-0x3000ff irq 70 on simplebus0
mmc0:  on sdhci_bcm0
uart0:  mem 0x201000-0x201fff irq 65 on simplebus0
uart0: console (115200,n,8,1)
vchiq0:  mem 0xb800-0xb84f irq 2 on simplebus0
vchiq: local ver 8 (min 3), remote ver 8.
pcm0:  on vchiq0
bcm283x_dwcotg0:  mem 0x980000-0x99ffff irq 17 on simplebus0
usbus0 on bcm283x_dwcotg0
fb0:  on ofwbus0
fbd0 on fb0
VT: initialize with new VT driver "fb".
fb0: 656x416(656x416@0,0) 24bpp
fb0: fbswap: 0, pitch 1968, base 0x1e08c000, screen_size 818688
cryptosoft0: 
Timecounters tick every 10.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 480Mbps High Speed USB v2.0
bcm2835_cpufreq0: ARM 700MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF
ugen0.1:  at usbus0
uhub0:  on usbus0
mmc0: CMD8 failed, RESULT: 1
mmcsd0: 513MB  at mmc0 41.6MHz/4bit/65535-block
Trying to mount root from ufs:/dev/mmcsd0s3a [ro]...
warning: no time-of-day clock registered, system time will not be set accurately
uhub0: 1 port with 1 removable, self powered
ugen0.2:  at usbus0
uhub1:  on usbus0
uhub1: MTT enabled
uhub1: 3 ports with 2 removable, self powered
random: unblocking device.
ugen0.3:  at usbus0
smsc0:  on usbus0
smsc0: chip 0xec00, rev. 0002
miibus0:  on smsc0
ukphy0:  PHY 1 on miibus0
ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ue0:  on smsc0
ue0: Ethernet address: b8:27:eb:f1:62:e6
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Growing root partition to fill device
GEOM_PART: mmcsd0s3 was automatically resized.
  Use `gpart commit mmcsd0s3` to save changes or `gpart undo mmcsd0s3` to revert them.
mmcsd0s3 resized
mmcsd0s3a resized
super-block backups (for fsck_ffs -b #) at:
 556192, 667424, 778656
/etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Setting hostuuid: 901c1969-a9d5-11e5-996d-b827ebf162e6.
Setting hostid: 0x45acda89.
No suitable dump device was found.
Starting file system checks:
/dev/mmcsd0s3a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mmcsd0s3a: clean, 181490 free (26 frags, 22683 blocks, 0.0% fragmentation)
/dev/mmcsd0s2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mmcsd0s2: clean, 32726 free (6 frags, 4090 blocks, 0.0% fragmentation)
Mounting local file systems:.
Setting hostname: nanobsd-rpi-b.
Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy:eval: cannot create /boot/entropy: Read-only file system
smsc0: chip 0xec00, rev. 0002
ue0: link state changed to DOWN
ue0: link state changed to UP
Starting Network: lo0 ue0.
lo0: flags=8049 metric 0 mtu 16384
 options=600003
 inet6 ::1 prefixlen 128 
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
 inet 127.0.0.1 netmask 0xff000000 
 groups: lo 
 nd6 options=21
ue0: flags=8843 metric 0 mtu 1500
 options=80009
 ether b8:27:eb:f1:62:e6
 media: Ethernet autoselect (100baseTX )
 status: active
 nd6 options=29
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Starting devd.
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Starting dhclient.
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5
DHCPOFFER from 10.0.0.5
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 10.0.0.5
bound to 10.0.0.65 -- renewal in 3600 seconds.
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Generating host.conf.
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Creating and/or trimming log files.
Starting syslogd.
Setting date via ntp.
24 Dec 01:06:34 ntpdate[457]: step time server 204.9.54.119 offset 2122.146331 sec
Starting casperd.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Configuring vt: blanktime.
Starting cron.
/etc/rc.conf: 1.freebsd.pool.ntp.org: not found
Starting background file system checks in 60 seconds.
override rw-rw-r--  730/730 for /firstboot? 

Thu Dec 24 01:06:47 UTC 2015

FreeBSD/arm (nanobsd-rpi-b) (ttyu0)

login: root
Dec 24 01:06:50 nanobsd-rpi-b login: ROOT LOGIN (root) ON ttyu0
FreeBSD 11.0-CURRENT (RPI-B) #0 r292645 0ff2d15e64f9+ qbase/qtip/rpi/tip: Wed Dec 23 17:29:02 MST 2015

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
root@nanobsd-rpi-b:~ # mount -uw /
root@nanobsd-rpi-b:~ # rm /firstboot
root@nanobsd-rpi-b:~ # uname -a
FreeBSD nanobsd-rpi-b 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r292645 0ff2d15e64f9+ qbase/qtip/rpi/tip: Wed Dec 23 17:29:02 MST 2015     imp@zooty:/usr/home/imp/FreeBSD/rpi-b/obj/arm.armv6/usr/home/imp/FreeBSD/head/sys/RPI-B  arm
root@nanobsd-rpi-b:~ # uptime
 1:07AM  up 1 min, 1 users, load averages: 0.87, 0.39, 0.16
root@nanobsd-rpi-b:~ # ifconfig
lo0: flags=8049 metric 0 mtu 16384
        options=600003
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21
ue0: flags=8843 metric 0 mtu 1500
        options=80009
        ether b8:27:eb:f1:62:e6
        inet 10.0.0.65 netmask 0xffffff00 broadcast 10.0.0.255 
        media: Ethernet autoselect (100baseTX )
        status: active
        nd6 options=29
root@nanobsd-rpi-b:~ # ntpdate 0.freebsd.pool.ntp.org
load: 0.48  cmd: ntpdate 671 [select] 8.16r 0.05u 0.04s 0% 3112k
24 Dec 01:07:44 ntpdate[671]: adjust time server 69.164.201.165 offset 0.001943 sec
root@nanobsd-rpi-b:~ # exit
logout

FreeBSD/arm (nanobsd-rpi-b) (ttyu0)

login: 

For Great Justice

I/O was beginning
Kernel: What Happen?
Driver: Someone set us up the DMA.
IO APIC: We Got Signal
Kernel: What?
Driver: IO MMU Turn ON
Kernel: It's You
VIRTIO: How are you Memory!
All Your Bytes are belong to us
You are on the way to destruction
Kernel: What Say You?
VIRTIO: You have no chance to survive
Make your DATA
HA HA HA HA
User: Kernel?
Kernel: Take off every BIO
You know what you are doing
move BIO For great Justice!

Simple DirecTV Hack to get Netflix without another ethernet drop

DirecTV has been using MoCA for some time to implement their whole home DVR. I've been a DirecTV subscriber for years. In my previous house, I had easy access to the locations I had the TV and it was trivial to pull the Cat5 cable to those locations. In my new house, however, a number of issues made it tricky to pull new cable. The finished basement sealed in many of the cable runs without a nice pull-string. The cables were run through the joists in a point-to-point manner. The exterior walls of the house are insulated with hay bales. These are great for insulation, but impossible to run electrical cable though (and even if it wasn't, fire codes prevent it). Finally, the attic has lots (48") of blown in cellulose insulation. Great for my heating bills, but not so good for access.

I went ahead and had my DirecTV professionally installed. As a life long DIY person, this was tough for me. However, time was short and this saved me a day of dinking with it. The installers have gotten much better since some bad experiences I had 15 years ago now. They got me all setup. I have a nice new HR24 in the main TV area, and a HR23 from my old house. Sadly, I had to let go of the HR10 which I'd been pulling video off of for years. Such is the price of progress.

In the basement, they installed a DECABB1MR0 (Direct TV Ethernet to COAX adapter). This is a simple device that connects to the cable plant coming off the SWM to your ethernet. They connected this so the HR23 and HR24 would be connected to the internet. This technology is called MoCA, and it drives much of the home sharing. The HR24 has MoCA built in, but the HR23 didn't. The cable guys installed the HR23 directly into a slightly different device (a DECA-II that was powered from the HR23 box and provided a single ethernet port). They told me it was the only way it could work. However, they were mistaken. I now have a ethernet switch with the DECA-II, my HR23 and my TV with Netflix. This works great. It is a bit anti-climatic, though, since I also had a cat5 port near this TV I could have used instead. But since I needed the router anyway, I thought I'd save myself some time and trouble troubleshooting the Cat5 port (all of them in this house have been wired wrong) and use this setup. I've watched movies from Netflix, DirecTV shows, shows recorded on my HR-24 as well as on-demand shows streamed over the internet. It's all good.

Next, up, I had to tackle the HR-24. There's no power for the deca-ii adapters coming off this box. And it didn't occur to me until I was writing up this blog post I could have just used a deca-ii by switching the port in the SWM from the unpowered to powered port and plugged the ethernet directly into my sony player. However, I was able to do the next best thing. I got a second DECABB1MR0 on ebay for $10.00 or so. I got a good 2-1 splitter (3.5dBm loss) and some good 1' RG-6 cables online. I was then able to wire the cable from the wall into the splitter. One leg went to the HR-24. The other leg went to the deca to go onto my sony player. The splitter did result in a small loss of signal to the HR-24, but all the signal strength levels look good on the setup screen. I've been able to stream Netflix off the Sony DVD player as well. I've not yet seen how well this responds in snow and ice.

Anyway, just a simple connection of different technologies. I learned these devices are nice MoCA bridges, and you can get decent performance over MoCA (decent enough for streaming video), but that the ping times are much greater (3-4ms rather than 10-100us that I'm used to seeing over GB ethernet). The technology is a bridging technology, with multiple devices allowed on either side of the bridge. The 3dB splitter worked better than I feared it might and is a simple way to connection. The powered port option should be kept in mind for future expansion if and when I add another receiver.

Fixing FreeBSD/arm in stable/9...

Just a quick note. I recently tried to create a FreeBSD-stable 9 image for an Atmel AT91SAM9G20 board I have. It didn't work. So, I tracked down the problems. It turns out that when the unmapped BIO changes went into stable/9 in revision r251874 a single revision (r246881) was missed. This left the ARM busdma API effectively non-functional. Unfortunately, this merge happened before the 9.2 release, so it is quite likely that all platforms of FreeBSD/arm didn't work in 9.2-RELEASE. I've not confirmed this, but the Atmel board I have wouldn't have working networking (and quite likely working USB) in 9.2-RELEASE. A few revisions later, the VM layout changed slightly. This also broke Atmel in a different way. I've corrected both of these issues and as of r259093, and I have my Atmel AT91SMA9G20 board working, at least mostly. There still appears to be an mbuf memory leak in the ate driver that takes the system out after a few hundred megabytes of traffic. This makes it less than completely useful in NFS root operations. My ultimate goal isn't an NFS root, but it is an network server, so I guess I better track the root cause of this down...

Tracking down the problem...

OK. After fighting several gdb build issues, I gave up on earlier versions. I thought I'd go old school to try to find the problem. First step was to figure out where we were dying. I had only one clue, the crash location form /var/log/kernel:

Nov 24 19:12:02 (none) kernel: emulate_load_store_insn: sending signal 10 to fred(8249)
Nov 24 19:12:02 (none) kernel: $0 : 00000000 9001fc00 ffffffff 0000001f 9fff7b3c 7fff7b40 00000000 00000000
Nov 24 19:12:02 (none) kernel: $8 : 0000fc00 7fff7b44 00000000 ffffffff 80151b78 000067b2 000033d9 80184020
Nov 24 19:12:02 (none) kernel: $16: 0040bb00 7fff7c44 004015c4 00000003 00407ccc 1002340c ffffffff 100267ac
Nov 24 19:12:02 (none) kernel: $24: 00000000 020cfe48 1000b5f0 7fff7aa0 7fff7aa0 0040899c
Nov 24 19:12:02 (none) kernel: Hi : 00000000
Nov 24 19:12:02 (none) kernel: Lo : 0000004e
Nov 24 19:12:02 (none) kernel: epc : 00408b70 Tainted: P
Nov 24 19:12:02 (none) kernel: Status: 8001fc13
Nov 24 19:12:02 (none) kernel: Cause : 00000010
Nov 24 19:12:02 (none) kernel: 8001e9fc 8001eac0 80022bb4 800215b4 8001d6b8 00408a7c
Nov 24 19:12:02 (none) kernel: 00408a7c 0040899c

We know that epc is the address of the faulting PC in this message. So we died at 0x00408b70. Normally, gdb would tell us where this location is. Something is busted with stabs in gcc 3.0 and 3.3, so I couldn't get a good traceback when we died. So, I had to go old school. First, I disassembled the code:

mips-TiVo-linux-objdump -S -d tivovbi > tivovbi.dis

So I needed find 0x408a7c. Looking for it, we find:

408b64: 8c420000 lw v0,0(v0)
408b68: 00000000 nop
408b6c: 3043001f andi v1,v0,0x1f
408b70: 8c820000 lw v0,0(a0) ----- here ----
408b74: 00000000 nop
408b78: 00621007 srav v0,v0,v1
408b7c: 30420001 andi v0,v0,0x1
408b80: 1040007f beqz v0,408d80 main+0x1054
408b84: 00000000 nop
408b88: 8f99822c lw t9,-32212(gp)
408b8c: 00000000 nop
408b90: 0320f809 jalr t9
408b94: 00000000 nop

Sadly, this isn't as helpful as I'd like. There's no STABs interleaved, nor any source listed. This is less useful than I'd hoped, and likely the reason that gdb can't cope either. So what to do? Have gcc tell me the raw assembler that it is listing...

mips-TiVo-linux-gcc -g -DTIVO_S2 -c tivocc.c -S > tivocc.s

So, now we have to look for the 'lw v0,0(a0)' instruction and hope for the best. Trouble is, gcc outputs raw register numbers, so we have to lookup that in OABI v0 is $2 and a0 is $4. So, we have to look for lw\t$2,0($4) in the output:

$LM506:
.stabn 68,0,930,$LM506
lw $2,76($fp)
srl $2,$2,5
sll $3,$2,2
addu $2,$fp,160
addu $4,$2,$3
lw $2,76($fp)
andi $3,$2,0x1f
lw $2,0($4) ----------- here
sra $2,$2,$3
andi $2,$2,0x1
beq $2,$0,$L348

There's other places where this instruction is used, but this the only place where andi comes in the immediate instruction before the lw. The rest matches tolerably well. So we've found where we started. It turns out .stabn structure 930 is the line number. This leads me back to the following line:

if(FD_ISSET(remote_fd, &rfds))

How can this be wrong? It turns out it isn't wrong. Elsewhere, remote_fd is overwritten. But this turns out to be due to a bug in gcc 3.0. When I updated to gcc 3.3, the crash goes away. It turns out that

if (flags & TEXT_MODE && FD_ISSET(0, &rfds) && (ret = read(fileno(stdin), &c, 1)) == 1 && c == 'q')

is compiled such that it corrupts remote_fd. When parens are put around the first bit, the problem also seems to be fixed. gcc 3.3 compiles both of them identically. This has ended unsatisfyingly, since I don't see the bug in the assembler (so maybe it just moves the actual bug such that it doesn't bite this variable). Despite the odd end, I thought I'd share how I traced down what line this was at, in case others could benefit from the techniques.

Cross building gdb for TiVo series 2. First attempt fails to work.

OK. Flush with pride after the success of building tools for my TiVo, I thought I'd get the program that I built the tools for working. It isn't obvious what's going on, so I thought I'd build the debugger to look at the core files that I'd been able to get.

So, first the build steps:

tar xvf gdb-7.6.1.tar.bz
cd gdb-7.6.1
rm -rf intl texinfo/ sim
mkdir build-gdb
cd build-gdb
../gdb-7.6.1/configure --target mips-TiVo-linux --prefix ~/tivo/tools/mips-tivo --with-lib-iconv-prefix=/usr/local --without-expat
gmake
gmake install

It took a little bit of dorking around to find all the details, but the FreeBSD gdb port gave some good hints.

But this versions must be too new :(. All the core files I was able to generate failed to get a good traceback.

So, I'll give it another shot with an older gdb.

Building old-school TiVo build tools on FreeBSD 9.2-stable

Intro

As long-term readers of my blog know, I've been nursing along a TiVo HR10-250 for the past few years. It works great for the low-quality material that my younger son loves to watch, plus I can harvest video off of it with ease.

Recently, we switched up how it was connected to our TV (and indeed, got a new TV too). During this process, we lost the ability to display subtitles. Since my wife and I like to watch The Daily Show and other similar shows upstairs without going downstairs to the big TV with the HD DirecTV player, and the environment upstairs can be a bit noisy, subtitles are quite useful.

So, after crawling around the DealDatabase forums for a bit, I found a good program called tivovbi. It works really well for displaying closed captioning. However, I used a binary I found on the forum. Nothing is more annoying than a program you download from a forum that randomly core dumps for reasons that are totally mysterious.

I can't even hack it to do anything since I have no TiVo build tools.  Looking for tools online, I can't really find anything that isn't just a Linux binary.  The Linux binaries have issues with the FreeBSD linux ABI implementation, owing in large part to their age (binaries from 10 years ago have some issues, I think with just the packages are needed having a subtle incompatibility).

So what should I do. I could create a virtualbox VM and run linux. But then I'd be running Linux, and copying back and forth to a VM is always a hassle in some way.

So, the other alternative is to build the tools from source. I thought it would be easy to do this, but there's a number of issues with it, so I thought I'd write up my experience. This is on a FreeBSD 9.2-stable system on amd64. That last bit will turn out to be important in a bit, because nothing was easy and simple on this project...

After reading through these instructions, I can't help but marvel at how this lack of integration is tolerated, but to be fair, this is building tools that are nearly a decade old at this point and I did have to kludge around lack of support for 64-bit x86 in gcc... This is nearly 60 separate commands to type. Yuck. Also, looking at the layout in chrome, many of the line breaks have liberties taken with them, so your best bet may be to cut and paste much of what I'm doing here...

Locating the tools

Tivo Utils has a bunch of useful links. Including the linux binaries that I've had issues with. Thankfully, there's a shell script called 'build_mips_x_compiler.sh' which builds all the tools and the basic libraries. I thought I could just run it and have everything built. As with everything else in this project, this wasn't so much the case. So I wound up doing a lot of things by hand.

The first bit of the script fetches all the source tar balls. The gnu stuff has had long-term stable paths, so they fetched. However the TiVo linux didn't transfer, since it was in a new location. In fact, it was also for 4.0, and my TiVo was running 6.4a. So, I had to grab that from TiVo linux downloads page. The Linux 6.4 download was exactly what I needed to grab. Once I had these in place, I was ready to start building. You also need binutils 2.13, Gcc 3.0 (yes, 3.0!), glibc 2.2.3, and glibc 2.2.3 linuxthreads support.

Also, I'm installing all the tools in ~/tivo/tools tree, and building them from ~/tivo/toolchain. Fetch all of the above linked tarballs into ~/tivo/toolchain (or whatever you want). Now that we've found the tarballs, we can start with the builds. I'll assume the following environment variables are set. TARGET is set to "mips-TiVo-linux" and PREFIX is set to "$HOME/tivo/tools". I've also added ${PREFIX}/bin to my PATH (I didn't from the start, and got quite far before it mattered). I also had gnu make installed as gmake from ports.

Building Tools

Binutils 2.13

Binutils 2.13 is almost trivial to build:

cd ~imp/toolchain
tar xvfz binutils-2.13.tar.gz
mkdir build-buinutils
cd build-binutils
../binutils-2.13/configure  --target=$TARGET --prefix=$PREFIX
gmake -j 20 all
gmake install
cd ..

gcc 3.0 stage 1

gcc 3.0 stage 1 took a lot of trial and error to create. Turns out, there's no support for FreeBSD 9 in this tarball. That's trivial to add, and I've provided some patches. However, the next problem is that FreeBSD/amd64 isn't supported. This problem turns out to be more difficult to overcome. So you have to configure for FreeBSD/i386 (or backport amd64 support, which I thought would be too hard, so I didn't do it). This means you need to have a 32-bit compiler. After all the talk about making cc -m32 working, I thought I could just do gmake CC="cc -m32". This failed, however. The 32-bit applications dumped core, meaning that gcc couldn't complete its bootstrap process. I wound up having to use FreeBSD's xdev to do it:

pushd ~/FreeBSD/svn/stable/9
sudo make xdev XDEV=i386 XDEV_ARCH=i386 WITHOUT_CLANG=t
popd

Once we have that in place, we can build gcc. Except that it doesn't build quite right. We need to add some additional patches. One to work around a weirdness in the obstack implementation where it relied on an illegal lvalue autoincrement, and one to work around crazy sh that's likely bogus in fixincl. All of these are included in the above patch.

mkdir build-gcc
cd build-gcc
../gcc-3.0/configure --target $TARGET --prefix $PREFIX --without-headers --with-newlib --disable-shared --enable-languages=c --host i386-foo-freebsd9
gmake CC=/usr/i386-freebsd/usr/bin/cc all-gcc
gmake CC=/usr/i386-freebsd/usr/bin/cc install-gcc

If you wanted to see if -m32 worked, you could skip the make xdev step above and substitute CC="cc -m32" in the two gmake commands. I've also tested values up to 20 for -j for at least the first command.

Building the Linux Kernel headers

The following comes pretty much verbatim from the build_mips_x_compiler.sh and have been verified to work. You'll need to grab this patch for these instructions to work. It works around an expr difference, as well a really cheap kludge to get autoconf.h generated. Also, I had to install the bash port, and create a symlink from /bin/bash to /usr/local/bin/bash, which I've not put inline...

tar xf TiVo-6.4-linux-2.4.tar.gz
cd linux-2.4
patch -p0 < ../tivo-linux-2.4.diff
yes "" | gmake ARCH=mips CROSS_COMPILE=mips-TiVo-linux config
gmake ARCH=mips CROSS_COMPILE=mips-TiVo-linux include/linux/version.h
mkdir $PREFIX/$TARGET/include
cp -rf include/linux $PREFIX/$TARGET/include
cp -rf include/asm-mips $PREFIX/$TARGET/include/asm

which will be enough to get us to the next step...  Woof, maybe I should just make a script for all this stuff... With so many fiddly bits, I'm not sure that's a good idea.

Building gmake 3.80!

Newer versions of gmake don't grok the Makefiles that glibc 2.2.3 generates. All kinds of weird errors and odd behavior. So, to make progress, you'll need to build gmake.

fetch ftp://ftp.gnu.org/gnu/make/make-3.80.tar.gz
tar xf make-3.80.tar.gz
cd make-3.80
./configure
make
cp make ~/bin/gmake380
cd ..

I didn't bother installing it, since I just need it for this diversion so I copied into my bin dir, that I have in my path.

Building glibc 2.2.3

Now we're on to glibc. Things have been smooth sailing up to this point. With glibc, we have to extract, configure, build, fix the build oops, build again, fix some info files, then install. Woof! Sure makes for a difficult to reproduce experience. And those are the build on linux instructions... Apparently there's a lot of host leakage that's making things somewhat difficult to reproduce... but so far that appears to be only with gmake 3.82. gmake 3.80 works much better. More patches needed.

tar xf glibc-2.2.3.tar.gz
tar xf glibc-linuxthreads-2.2.3.tar.gz -C glibc-2.2.3
env CC=mips-TiVo-linux-gcc ../glibc-2.2.3/configure --host=$TARGET --prefix=$PREFIX --disable-debug --disable-profile --enable-add-ons --with-headers=${PREFIX}/${TARGET}/include
gmake380
sed -i.old -e 's/elf32-bigmips/elf32-tradbigmips/' elf/rtld-ldscript
gmake380
gmake380 install
cd ..

Munging things around before the second assault  on gcc for its stage 2

Don't know why the original instructions take this time rearrange the deck chairs, but it seems to be necessary.

mv ${PREFIX}/${TARGET}/include/asm ${PREFIX}/include
mv ${PREFIX}/${TARGET}/include/linux ${PREFIX}/include
rm -r ${PREFIX}/${TARGET}/include
rm -r ${PREFIX}/${TARGET}/lib
ln -s ${PREFIX}/include ${PREFIX}/${TARGET}/include
ln -s ${PREFIX}/lib ${PREFIX}/${TARGET}/lib

Build gcc 3.0 stage 2

Now that we have everything else setup, it is time to build the second stage of gcc. Alas, gcc 4.2.1 doesn't allow constructs like (a ? b : c) = d, which gcc 3.0 uses to implement C++. So, no g++ for me. If you want g++, you'd have to build the gcc34 port... The rest is almost boring after all the other hoops we jumped through:

mkdir build-gcc2
cd build-gcc2
../gcc-3.0/configure --target=$TARGET --prefix=$PREFIX --enable-languages=c --host=i386-foo-freebsd9
gmake CC=/usr/i386-freebsd/usr/bin/cc all
gmake CC=/usr/i386-freebsd/usr/bin/cc install
cd ..

Building stuff

Now, it is time build things...  I'll write more blog entries about that...

How to fix a Washing machine

Let's say you have one of those fancy washing machines with sophisticated circuitry.

Well, then if your machine is like mine, it sucks to be you.

Last night, before a trip, the Kenmore Oasis HE machine that we've had for many years started doing something weird. You'd turn it on, and it would start cycling from Heavy, to medium, to light, etc on the soil setting. It wouldn't stop doing it. And if you somehow got it to pause for a moment, the cycle would do weird things: spin at the wrong times, drain at the wrong times. And if you left it on overnight, it would suddenly spring to live at 1am just as you were falling asleep.

So, new parts for this would never arrive in time for me to install them before the trip.

So, what to do. I tried all the obvious fixes: reseat all the connectors. Do it again just in case. Get some rubbing alcohol and clean off the accumulated gunk on the board. None of these fixed the problem. So looking at the board, I thought "geeze, why don't I just remove the switch from the circuit: maybe it's gone bad and shorted out." I mean, if you're falling to earth, you might as well try to fly, right? The switch was wired into the circuit through a diode. Since the diode was SMT and the switch was through hole, I decided to desolder the diode (D59 in the pic). I had no rational basis for thinking this would work. Other folks have reported exactly this same problems on different washing machine forums, but the solution was always "start replacing boards" which wouldn't fly for me today.

So far, it seems to be working. I hope this repair will hold until I get back from my trip...

This photo shows my test run, where I just removed one side....

Who said these new washing machines are irreparable... Oh wait, all the repair guys that have come out and all the online forums, and this isn't really a fix so much as a kludge until I can get back to fix it right...

What could be so hard about an external toolchain.... Step 1: How big a problem...

FreeBSD is moving from gcc to clang as its base compiler. This works really well for x86, and moderately well for arm. However, it doesn't work very well for any other architecture that FreeBSD supports.

Brooks Davis recently committed changes that he claims allow him to build FreeBSD/mips with an external clang and the cross-binutils port.

I thought "how hard can that be, I'll knock together a port for gcc that lets me use that instead of the native compiler.

So, with a little bit of svn foo, I was able to extract the differences in FreeBSD's gcc from the stock 4.2.1 gcc that it is based on. I thought the differences would be small and manageable. However, they were large. In excess of 10,000 lines in the diff file.

Undaunted by this large size, I started breaking down the diffs. Well, first, the diffs, excluding the docs, are 10,652 lines in size. I didn't save how big they were with the docs included.

So I went to separate out the wheat from the chaff. First up is 5172 lines of changes to x86. Why do we need that many changes to x86? Well, to support newer x86 hardware models. since I assume that will be in a newer version of gcc, I chose to totally ignore it unconditionally.

Next, I discovered 924 lines of changes for the FreeBSD configuration of gcc, plus various tweaks to gcc to make it the FreeBSD native compiler. Half of these seem relevant, the other half not so much (I'll discover how much later).

There's also about 2187 lines of bug fixes that are split between back-ports of gcc bug fixes and various warning cleanups to clang will be more happy. Add to that another 340 lines of g++ modernization...

So we're down to less than 2k lines of changes here. They break down as 220 for arm, 675 for mips, 220 for ia64, 245 for sparc and 350 for powerpc (total of 1910) and another 78 line for threading tweaks, 135 for code to check FreeBSD's kernel's printf extentions, and 130 for some tweak to assembler output that I don't quite get, so it goes into the 'look at it later to see if it is still needed' pile.

The first 4 patches slotted into the gcc 4.5 nicely (config-guess changes, the config + arm + mips changes). After tossing out all the extra goo we had been dragging around in our tree, they are about 850 lines of diffs. After adding in the other architectures, I suspect this will double. Plus the other relevant gcc changes, we may see as many as 2000 lines of patches for any port that could be used as a system compiler for any supported architecture).

I have the beginnings of that port, but it is early days. It builds, but still can't make it through buildworld let alone buildkernel. The external toolchain support needs a lot of tweaks to make it happy with gcc, and there's still much I don't understand about why some of them are necessary.  But I'll save those for another time.

Anyway, that's enough for now. I'll report also on moving these patches to 4.6, 4.7, 4.8 and 4.9 when  get around to it. I'll also see what it takes to move them into upstream sources. That should have been done years ago, but that's another tale of woe I don't have time for just now, and even if I did I'm not sure that dirty laundry would be appropriate for my hacking blog.

How C can bite you, an example

Consider the function foo:

int foo() { if (error) return -1; else return 12; }

and consider its use here:

int n;
...
if ((n = foo()) >= sizeof(bar)) { memcpy(dst, src, n - sizeof(bar)); }

Seems simple and safe right? Well, not so much.  The 'C' standard states that when you have types that aren't of the same rank, a conversion must happen. So, since n is an int, and sizeof(foo) is a size_t (unsigned long), a weird conversion happens when you hit an error in foo. It returns -1, but that -1 is converted to MAX_ULONG, so the conversion is true, and we copy way too much memory smashing everything in it's path.

When a coworker asked me about this, I replied "And verily, thou shalt not compare an int to a size_t when thine int can be that which represents a plurality of deficit as well as a plurality of surplus." because it is easily clearer than the C standard...

TivoWebPlus 2.1 changes

As you may know, I still have a HR10-250 in service.  I run TivoWebPlus 2.1 on the box. It mostly works the way I want it, although at times it is a bit slow.

I checked out the latest version from the sourceforge web site many months ago and found there's some issues with it that I'd like to fix.

I'e made the following fixes:

1. Sadly, I didn't document very well where I got this fix from. It makes Movies show up as Movies and not as 'Not an Episode' in the ToDo and Now Playing fields. This appears to be a combination of my hacking, but mostly the work of "DJL". I think I downloaded it from a dealdatabase or other tivo forum posting, but I can't find it now.
2. Fixed the long series number problem so episode numbers are properly displayed.
3. Added a Thumb link so you can put those in your ToDo lists. I find this handy for quickly killing those things that Tivo thinks I might like, but I don't.
4. Fixed a problem with descriptions being nothing but spaces (this may have also been a forum find, but I can't find docs for that either)
5. Fixed a few problems with todofeed.tcl (not sure these are completely good, as I think I was trying to parse the output for something I did differently).
6. Fixed a spelling error.

OK, these changes won't change the world, but since all work has stopped on TivoWebPlus, I thought I'd share them with the world.  You can find my cumulative patches here. If anybody recognizes #1 or #4 or can locate their origin, please let me know by commenting

I've also hacked together a todo database publishing api. It is crude, but it allows me to cancel shows I've already downloaded via ftp and such.  On the off chance people find it useful, you can find it here.

If you find these things useful, please comment. Not sure how many people may be interested, or if the TiVo hacking world has totally moved beyond this device...

Some notes on HR10-250 video

Over the past 5 or 6 years, I've had my TiVo HR10-250 setup to extract videos off of it.  Please see my other posts on how I set that up with Zipper and mfs_ftp.  There are some newer programs to help with the video extraction, but I have a cobbled together system that works OK enough for now, so I'll not go into those here.  Today I'll explore how to convert the files mfs_ftp can produce into something that looks good, at least on the TV I have :). I'm in the US, so all these instructions are NTSC-centric. Sorry for all my pals out there that have PAL.

What I will talk about is converting the tmf files (or the .ty files) into something useful. For me, useful varies with time. The tmf files are like ty files, except they are big tar balls of a series of .ty files (all smaller than some limit that I can't recall) and a xml file that describes the show. The tytompeg program is what I use to convert them to a mpg file. You can find info on how to snag this here DVRpedia entry on TyToMpg.

The tytompg program produces video that is 480x480 with a 4:3 pixel aspect ratio.  This turns out to be the SVCD resolution, so many tricks for turning SVCDs into DVDs work when processing these files. However, they aren't fully compliant SVCD files. There's some audio delay issues that cause problems and many tools have difficulties with these files. I've only overcome the synchronization problems by trial and error at this point (certain ffmpeg operations causes something to remove the stutter, sometimes on some versions). One problem with tytompg is that I lose the subtitles. I'll explore that problem more later...

A number of cartoons are broadcast these days on the SD stations in letterbox. My son watches them. It would sure by nice if I could process these files to produce a nice, 16x9 videos that play nicely. Since these files are 480x480, the actual video is in the pixels from 8,60 to 472,360 (the extra 8 pixels is a band that many stations seem to put around the picture for reasons unknown). Sometimes there's only a few pixels on each side, so it isn't worth the hassle to get rid of them (in which case the picture is bounded by 0,60 to 480x360). 720x480 is the only DVD resolution that supports a 16:9 aspect ratio. Speaking of aspect ratios, please see the writeup at http://howto-pages.org/ffmpeg/ which covers this in extreme detail. Here's the ffmpeg 1.0.1 command line I use to convert them (the 0.x command needs to crop based on the 720x480 image size).

ffmpeg -i 002-ty.mpg -vf yadif,crop=464:360:8:60,scale=720:480 -aspect 16:9 -vcodec mpeg2video -s 720x480 -b 4500k 002-nice.mpg

mpeg2video is easy to create and fairly ubiquitous, but has a couple of draw backs.  First, it is large. The above command typically doubles the size of the file. It looks really nice on the big screen HD TV (so ffmpeg's upscaler is a little better than the upscaling in the TV), but eats up the storage space. Second, many of the media renders I have in the house do mpeg2 rendering in software, so the playback can be a bit jerky on those devices.

Converting to h.264 is an option that I'll explore in later columns. The current level of tool integration isn't so nice, so would be a much longer write up.

DirecTiVo HR10-250 restore

I had to move the hard drive between two TiVos recently.  Since it was on DirecTV, I got the dreaded Error #51.  Since I couldn't recall what I did last time I got it, I googled it, only to find I needed to reset the unit with a Clear and Delete Everything (C&DE).  Turns out that since I'd hacked the TiVo, I had a much easier option that I'd forgotten about.

Sadly, I cleared everything before I'd discovered this much easier option.  I'm linking the TiVo forum post from here so that my future self will look at it before I have to go through this pain again.  Good thing I saved all the hacks I'd done to the TiVo along the way...

Recovering from Error #51

Hopefully I can easily recover the steps. Zipper sure makes it easy. It also adds the Enhancement Script that's knocking around to do all kinds of useful things. Makes hacking the TiVo way too easy...  Normally I would scoff at this script, but it is a good aggregation of knowledge that's hard to come by elsewhere, except sifting through the detritus of three or four different TiVo forums.

Anyway, just random hacking away...  Maybe the newer direct tivos can be hacked so I can download HD data again...

Oh, and one other thing: The zipper iso that's created can be booted in virtual box. If you connect your hard drive via a USB to PATA adapter, then you can connect that to the same virtual box.  The trick I found here was to create a vmdk that pointed to the drive.  When it is /dev/disk1 on my mac, I just do "VBoxManage internalcommands createrawvmdk -filename ~/TiVoDrive.vmdk -rawdisk /dev/disk1" and then connect that using the Storage tab in the config dialog for the virtual machine in VirtualBox. Handy stuff that...