Extracting part of the FreeBSD tree

Extacting the history CTM into its own

In the early days of the FreeBSD project, CTM provided a competitive advantage to the project by allowing those that weren't completely connected to the internet. CTM provided convenient way to get the sources via mag tape or mail (sometimes over UUCP links which were popular at the time).

Recently, the notion of removing from ctm from came up. Maybe it's a good idea, maybe it isn't a good idea. I thought it would be nice to know how hard it would be to extract the history of ctm into its own repo. So, I set out to see how hard it would be. Turns out, git makes it easy.

Initial setup

First, clone a tree.

git clone https://github.com/freebsd/freebsd

Next, we need to remove everything except ctm:

git filter-branch --prune-empty --subdirectory-filter usr.sbin/ctm

This will leave a repo with the complete history and a ctm directory at the top level. We're done, right?

Well, not so fast. We actually aren't done. There's a lot of crap still in the tree. Despite saying to get rid of empty commits, there are empty commits. Most of them are from merges to the tree that didn't actually touch ctm, but were merged and git has all of those in the tree.

Rebasing to clean away the sins

So, how do we clean this up? By rebasing of course. So, to make things easy, I tagged the first version in the new repo with a tag. I used 'base'. I also tagged the tip of master after the prune with 'orgmaster' in case I needed to start over (which I've not detailed here, but I'll just say it came in handy).

git rebase -i base master

Now, if you try this, it won't work. All the merges and cvs2svn 'fixups' are a problem. After a bit of trial and error, I figured out the list of things to remove from the todo list. Then I learned there was an easier way to find this list

git log --merges

will show the merges that still remain in the tree. Remove them from the todo list and then let the rebase proceed. There's likely a clever shell-script that can do all this, but I've not written one.

Sanity check

The sanity check step is easy. Diff the checked-out ctm repo with the freebsd repo's usr.sbin/ctm tree:

diff -ur ctm freebsd/usr.sbin/ctm

and it should be identical.  If not, you need to redo prior steps until it is done.

Pushing the result

So I created a new repo on github (in bsdimp/ctm).

git remote add upstream https://github.com/bsdimp/ctm

And then push it upstream:

git push upstream master

and you should be done. https://github.com/bsdimp/ctm should now have a repo that you can look at and have it match your local tree.

Next Steps

let the pull requests come in :)

Arm testing

Testing before the branch

So, tonight I tested 4 images that were 12.0 snapshots. I downloaded all the arm images that the re@ produces. The version string was 20180719-r336479.

GUMSTIX is impossible. There's no hardware it could run on, so I ignored it. It has an GUMSTIX XSCALE kernel, but a new GUMSTIX Duovero uboot and armv4 user land. Complete no go.

WANDBOARD and RPI2 booted on their hardware. I was able to dd the images and put them into my existing test bed. They just worked (but I haven't done more than just test boot, so caveat emptor).

I recently bought a new Raspberry Pi 3 B. I had a couple of RPi 3 (v1.2). I couldn't get any of them to boot at all on the RPI3 arm64 image. No output at all. [Update: thinking it may be a bad uSD, so will try again]

I also tried my old Pandaboard. It didn't work: no valid DTB was found in ubldr. [update: after selecting a good DTB, I get a core dump attaching ti_sdma_attach when calling device_get_softc()\

Later, I tried the CUBIEBOARD config on an actual Cubieboard. It worked too.

[UPDATE] PINE64 worked on my PINE64 kick-starter board. Woot!

On a related note, it was a sad day. The RPi3 B replaced my old Atmel AT91SAM9G20 board. It's my last Atmel hardware that I've turned off (I still have a ton of old Atmel gear, and some newer armv7 A5-based ones, but none of it is powered on: FreeBSD never supported the new gear, and the old gear barely works anymore).

Recovering git repo

I had a crash shortly after updating my git tree to master. When I rebooted, git was confused:

% git status
fatal: not a git repository (or any of the parent directories): .git

Needless to say, my heart skipped a beat. I have a ton of un-backed-up branches in this tree. There's many things that can cause this, according to a quick web search. However, a college told me they've seen this and the most common problem is with .git/HEAD. Looking at mine, it was 0 bytes long. This file has pointer to the latest branch checked out. Since I'd just checked out master, I was able to fix this by creating .git/HEAD:

ref: refs/heads/master

and life was good.

 

How to get a memory mapped serial console

Memory mapped uart

So, a friend was bringing up FreeBSD on a new system. It didn't have a traditionally mapped UART, but a memory mapped one, like we have in the embedded world. After stumbling around, I thought I'd document how to get a serial port in a situation like this.

First, he had linux running. It recognized the serial port. And it worked in UEFI.

In Linux, we first found the ttys:

% dmesg | grep tty
AMDI0020:00:ttyS4 at MMIO 0xfedc9000 (irq = 3, base_baud = 30000000) is a 16550A
AMDI0020:01:ttyS5 at MMIO 0xfedca000 (irq = 4, base_baud = 30000000) is a 16550A
ttyS4 at MMIO 0xfedc9000 (irq = 3, base_baud = 30000000) is a 16550A
ttyS5 at MMIO 0xfedca000 (irq = 4, base_baud = 30000000) is a 16550A

So, this is weird. Or rather, it's normal if you are used to ARM s16550's memory mapped, or some PCIe, PCI and CardBus serial cards.

It turns out that even if we don't have a driver for this that attaches automatically (uart works, but needs some glue on this platform), you can have a serial port console none-the-less. At the boot loader OK prompt:

OK set hw.uart.console="mm:0xfedc9000,rs:2"

What's the "rs:2"? If you look at the source it's Register Shift. It means that the cadence of registers isn't 1, but 4. It also means that each register is 32-bits, but only the lower 8 bits are valid. That's also typical in ARM and MIPS embedded processors.

So now you know.

FreeBSD Lua Loader (started in GSoC 2014) about to land

Just a quick note. I've been working on making the /stand environment easier to deal with and integrate into. After months of work on and off, the using Lua as a boot loader extension language is ready to roll into -current.

A preview review can be found here.

Setting up a SVN mirror

Setting up the svn mirror

Just following the steps in

https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/subversion-primer.html

in the Setting up a svn mirror (5.4.7) but have found there's some issues...

First step, grab the svn seed file

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r290116.tar.xz

this takes about 30 minutes. Also grabbed svnmirror-ports-r400410.tar.xz for a ports mirror. These file names are different than the instructions.

Next extract this somewhere (say /home/svnmirror/base) and then update it with "svnsync sync file:///home/svnmirror/base" and then set it to updating. You can do similar things for docs and ports. These files are a couple of years old now, so the sync still takes a several of hours. I believe that this is due to the protocol round trips required to extract it being sensitive to the latency between my machines and the FreeBSD svn server.

I use the following shell script as a cron job to keep things up to date.

#!/bin/sh
URL=file://$HOME/svnsync
logdir=$HOME/svnsync/log
lock="lockf -s -t 0 -k /tmp/svn-sync-mutex"

${lock} svnsync sync ${URL}/base >> ${logdir}/svn-base.log 2>&1
${lock} svnsync sync ${URL}/ports >> ${logdir}/svn-ports.log 2>&1
${lock} newsyslog -r -f ${logdir}/newsyslog.conf

FreeBSD cumulative commit graphs

FreeBSD Commit Graph

I've updated an old graph from a talk I gave in 2010 to have data up through today. The rate of development seems fairly steady.

When last I did this release, stable branches were flatter and lived longer. Now there's a fair amount of activity after the branch that continues for years.

Note: The hash marks are generally releases. I've removed the 'artificial' commits that were needed as a result of the CVS to SVN migration the project did a few years ago. The 2.0.5 branch has been omitted, and the 2.2.9 release (never tagged in the repo and < 10 changes different than 2.2.8). I had release numbers on the hash marks in earlier revs, but that's too busy now.

Reading in my Rainbow Floppy Collection

Recently, I acquired a kryoflux board to help read in the Venix floppies... I finally have it up and running.

I've managed to read in most of my collection of ~300 Rainbow floppies. 95% of them were actual Rainbow disks. The other 5% were either completely blank, or some variety of IBM format (including 3.5" 720k on a 5.25" floppy!) with a couple of 'robin' formatted ones just to keep you on your toes. I didn't wind up seeing any of the very rate dual-sided RX-50's (they exist, but are super rare because this was a home-brew job not an official thing).

While it is possible just to do
dtc -ffoo.img -i4
It takes a long time since it's reading the back-side that has no data on it. Using
dtc -ffoo.img -g0 -i4
is all that's needed to go fast. Some caveats should be observed.

Nearly every track has extra data in at least one of the sectors. This threw me for a loop, but is harmless (as stated in the manual).

I had some issues with old, ill-maintained floppy drives. Bought a refurbished one and all my troubles just went away. I had several old drives and based on the good floppy drive, I could find known good floppy diskettes and was able to refurbish one of my old units. I had the best luck with the TEAC FD55-GFR drives, btw. Not sure it's worth while to refurb the rest of the units, though, since I have something that's working...

Keep an eye on the number of sectors reported. It's usually 10. If it isn't, stop right away. It isn't a Rainbow / RX-50 formatted disk. If it's 8 or 9 sectors, it's likely an 320k or 360k IBM floppy. Removing -g0 and adding -k2 is useful for these diskettes (saves time more than anything. If it's 15 sectors, it's likely a 1.2MB floppy. Had 2 or 3 of those mixed in with these diskettes despite never having a machine that produced them.

As these disks age, you may need to try multiple times. I have some disks, though (maybe 1%) that I can't read back certain tracks on even with that. The default is good, but trying more helps with a similar number (so -t10 -tc5). I don't know if there's a way to retry individual sectors, or if dtc is internally buffering good sectors from previous retires or not. Previous programs I've had on the Rainbow for extraction had great luck retrying sectors multiple times when the full track read didn't work...

Some disks are copy protected. Not sure how best to preserve these disks. The -i4 format won't work with, for example, Lotus 1-2-3 system disks because tracks 78 and 79 have sectors numbered up to 12 and are missing sectors (they do this with a special format that I believe omits sectors 9 and 10 and instead has sectors 11 and 12). There are other copy protected disks that have deleted sectors since the Rainbow could easily read those, but they were hard to create from standard interfaces. Don't know if -i2 images would help, or if I have to go all the way down to -i0 to ensure proper imaging. The copy protected disks, however, were all broken in the '80s, so preserving them isn't so interesting to me. There's also a couple of disks that have data on track 82, but it's marked as track 79 and it's unclear what this data is. I know there are several old formats that preserve this info (TELDISK and DISKIMAGE being two that I've run into most).

cpmtools is able to extract data from CP/M formatted disks (use dec_pro format). It understands all about the interleaving you need to do on tracks 2+ and can work on raw kyroflux images.
As for MS-DOS disks, I wrote a small utility to convert the physical layout to a logical one, and to (optionally) replace sector 0 with a standard boot record with the BPB in it (Rainbow floppies have Z80 boot code which has a different starting byte (f3) and doesn't have the tables expected. Doing this, both mtools and FreeBSD's msdosfs can copy files easily enough. But so far it's not reversible, so you can't (yet) use it to create floppy images. I'll be testing two tricks (just prepending the magic sector and appending sector 0 at the end) to see if that might help. I can post a pointer if there's interest.

Venix disks are a mixed bag as far as data extraction, but are otherwise boring: 80 tracks, 1 side, 10 sectors per track. Just had brief access to the distribution diskettes, mostly prior to having kryoflux (and the reason for buying it). Ironically, I couldn't get the kryoflux working due to the drive issues I discussed above before I had to return the distribution diskettes. But I was able to copy them natively on my Rainbow...

Finally, there's still 2 or 3 disks that I've been able to read in that I have no clue what the format is. They are kinda sorta CP/M, but also kinda-sorta MS-DOS from looking at the first two tracks, but neither quite works to decode them. Physically, they RX-50. They don't look like ODS-2 nor v7 unix file systems, but that's harder to know for sure. They are unbootable (tracks 0 and 1 are 0xe5, the erasure pattern for RX-50s).

My only real complaint is that it would be nice if dtc would spit out a summary at the end, including tracks unreadable. I have to keep an eye on the logs otherwise. Not a huge deal, to be sure.

v7 compilation for simple .c's in /usr/src/cmd.

After about 4 and a half hours of build time, I have results to share. There's 110 .c files in /usr/src/cmd. 85 of these compile and link (it's unclear if they run). 25 have a compile or link error:

• accton, sa (lack of acct in libc)
• ar, nm, prof, ranlib, size, strip (mismatch between NMAGIC/OMAGIC and ARMAGIC_X)
• arcv (obsolete conversion program written in old style)
• clri, dcheck, dump, dumpdir, icheck, mkfs, ncheck, quot, restor (inode differences)
• dmesg (message buffer issues?)
• getty (freaks out compiler)
• graph (missing symbols for curses and libm stuff)
• osh (old shell? Written in funky style)
• ps, pstat (proc definition differences)
• tc (silly compiler issue, trivial to fix, but I have not Tek4015 terminals)

So 10 classes of issues for the affected files. Not too terrible. But there's other issues that will need looking into. 'ld' compiles fine, but it works on pdp11 a.out files, for example. cc compiles, but doesn't have the venix specific switches. I've written a shell script to do this, not a Makefile, and the commands used aren't quite right. Venix appears to support different amounts of stack under the data segment (but without docs, it's hard to know what flags I need), but none of my binaries have those. There's NMAGIC and OMAGIC binaries that have different meanings, and so on. All this makes the 'file foo /bin/foo' differ. Plus I haven't done the more complicated commands, nor have I poked at the compiler apart from cc, nor the kernel really at all, nor the libraries, nor the .y files. And factor.s and prime.s are pdp-11 assembler, so I'd have to pull those in from somewhere.

But given that almost all the normal commands compiled just fine, and many appear to work gives me hope that I might be able to reconstruct the sources used to build at last parts of the system. For some reason, that's appealing to me, but I can't really articulate a reason why, or come up with why that might be useful. Though it seems like a fun background project to celebrate the 40th anniversary of Version 7 in a couple of  years. Though running it on a 35 year old computer at that point may limit its appeal...

Rebuilding v7 sources on Venix, early results

I just started rebuilding the v7 sources on my Venix Rainbow on a lark. I'll post a full writeup, but here's a two early things I've learned.

First, it takes 1 minute to compile cat.c into cat. Yes, one full minute. On my normal server, it takes .1s of elapsed time. This is a 600x speedup since then, likely more since clang isn't known for being gentle on system resources.

Second, some compiler error messages are classic:

"getty.c", line 12: compiler error: insane structure member list

I have always thought getty to be rather opaque to use. The code is somewhat simple, but that simplicity doesn't lead to easy understanding in this case.

Most things not dealing with the filesystem are building w/o a problem. This is a pleasant surprise. Though the speed at which they are building isn't going to set any records. About 5% have issues that prevent compilation or linkage out of the box, though the problems look easy to fix. I started the build at 9:30. It's now 2 hours later and we're to building 'grep'. We're on pace for a rebuild of just the trivial foo.c -> foo programs taking 5-6 hours. This isn't for the more complicated programs like make or tar that have their own subdirectory.

Rainbow Venix with BSW Enhancements

I've taken some time to look at the Venix disks. I've managed to read them all after retrying the read errors enough. I've created new disks and installed it on my system. I'll go through all that in a forthcoming post since there's lots of fiddly bits.

This post I'll expose what BSW is. It's the Boston Software Works. It was a company that was run by Larry Campbell from 1985 until 1995. They had a number of different products over the years. Through about 1988 they offered their enhanced version of Venix for $800 a copy (not their choice on the price).  It was normal Venix, only better. Larry posted the following to mod.newprod on Usenet back in December1986 describing what it was. I've edited the formatting a bit, and trimmed the headers:

The Boston Software Works (BSW) is now distributing an improved version of VENIX/Rainbow.  VENIX/Rainbow is a complete UNIX(tm) system for the DEC Rainbow 100, developed by VenturCom, Inc. of Cambridge, Mass. It is a V7-based system, with extensions such as plot(3) drivers for the Rainbow color graphics board and DEC LA50 printer, semaphores, and shared memory.

Standard VENIX/Rainbow requires and supports the DEC RD51 winchester disk (10MB) only.

The Boston Software Works enhancements to VENIX/Rainbow include:

1. Support for any disk physically connectible to the Rainbow. Our development system runs a 70MB Micropolis disk.
2. Virtual consoles, as found in VENIX on IBM-compatible machines, and also in XENIX and Microport System V/AT.
3. Support for the NEC V20 chip, which replaces the Intel 8088 for a 10-15% performance gain.
4. Support for simultaneous use of a monochrome display (for terminal/console use) and a color graphics display (for graphics).
5. Ports (on an as-is basis) of several public domain programs, such as the Usenet news software, rn, smail, and Jove, an excellent EMACS subset.
6. Support for the CHS dual-winchester controller.

The price (single copy) for VENIX/Rainbow is $795 and includes a complete manual set.

I've installed both the original Venix/Rainbow and this enhanced version. The enhanced version is the way to go. It knows how to cope with the larger disk I installed in my Rainbow, and the installation process was much less painful (though there were some up-front gotchas that I wound up having to re-run WUTIL to get through).

These are all cool things. The only issue that I've seen is that the .o's aren't included so it's impossible to rebuild the kernel with these neat, new features without significant hassle. The other problem is that while I can get stable file transfer on MS-DOS with LCTERM at 9600 baud, I can't with Venix. 4800 is error prone even. Messing with the serial chip seems to also trigger the dreaded Interrupts Off message, so I've not messed with it too much. Still don't know if that's my Rainbow going bad, or other issues (have some spare parts arriving soonish). The maximum throughput is only 240-300 CPS anyway with LCTERM, so maybe I'll just run at 2400 or 3600 baud.

Not listed above: it kinda seems like it is reading my clikclok. I did have to fix the 'date' command to grok Y2k though (well, since I don't have the sources, I used the Unix v7 sources from TUHS). If this keeps up, I'll put together a distribution of useful things from V7 :).

I'll blog more when I've run through another install and worked out how to boot in MESS (it has a bug when booting directly off the hard drive, but maybe the BOOT program will fix that).

[[ Edited May 8, 2017 to supply link to original mod.newprod article on google groups ]]

Rainbow 100 Windows 1.0 disks, redux

Rainbow 100 Windows 1.0 Disks

OK. I never thought I'd find myself writing a blog about Microsoft Windows of all things. However, that's what I'm doing.

A while ago, on the cctalk mailing list I mentioned that I had Windows 1.0 disks for the Rainbow. I didn't think anything of it until someone contacted me about it. I'd never looked at them in the 15 years they've been in my collection. You needed the Graphics Option card, and I don't have one so I just filed it away in the back of my brain. I never looked into the situation further.

So, it turns out that the publicly available Rainbow Windows 1.0 disks are really not quite right. They are a collision between the Rainbow 100 version, and the VAXmate version. And evidentially, neither quite work. This collection has been cargo culted around in the three decades since they became available, but they didn't work. it was well known, but the archives contained what they contained and successive generations of hackers tried, and failed, to disentwingle the resulting mess. The mess is still available at archives of archives of archives, for example at Classic CMP there's a copy of the "goodnight" archive which is an incomplete copy of the various BBS archives that existed in the last 80's and early 90's.

Sometime in 2001, Arun Welsh and I corresponded about the DEC Rainbow. I forget the details, but if memory serves, he had a ton of disks that he was looking to get rid of and wanted to see them find a good home. The details are frankly fuzzy after 16 years. I got these disks and took out a few of the interesting bits, but mostly it was a collection of files that were available elsewhere so I didn't look too closely at the collection. But DEC Windows 1.03 stuck in my brain for some reason.

Recently, that boast I made in cctalk generated some email requesting a copy. Since I had my Rainbow out and was messing with it, I thought I'd oblige them. Once upon a time, I could read Rainbow disks, but that was a dozen hardware iterations ago, so I fired up my old machine and read the files in. There's two sets. One was Rainbow MS Windows and was 7 disks. It contained what appears to be a copy of the Goodnight archives with the entertwingled set of files. The other was 4 disks of DEC Windows 1.03.

The other one proved more interesting. It was a different set of files. It came with a letter tucked into the one of the disks giving permission to upload the set of disks to any BBS (including the recipient's BBS, who wasn't Arun, I've forgotten his connection to this mystery). If there's interest, I can post this letter.

Early repots suggest this is a good copy. If they prove true, I wonder how hard it would be to get things corrected in all the online archives.

I've uploaded to a github repo I've created. These files are under 'decwin' while the putative copy of the goodnight archives is under 'rbwin'. I've also uploaded a copy of wutil 3.2 source and binaries. I plan on uploading the Venix binaries there as well, once I get things sorted out and post instructions on how to install them either on real hardware, or in an emulator.

Lining up

Turns out I didn't need to look for exotic formats to solve this problem.

I had physical copies of the disks with 10 sectors per track. Since I couldn't get tar to line up, I posited that there must be missing sectors. Turns out I was missing something, but it wasn't sectors. That turned out to be a dry well.

What's going on is actually a lot simpler. I managed to get Venix installed on my Rainbow (I'll write up how in my next blog, it's fun to login to my rainbow from upstairs while it's down in the basement).

On the Rainbow 100, under both CP/M and MS-DOS there's an interleave of 2, but no per-track offset. This is done by having the logical sectors ride on top of the physical. So, if I write logical sectors in the order 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, it gets put on the disk in sectors with the labels 1, 6, 2, 7, 3, 8, 4, 9, 5, 10. Why go through this crazy arrangement? For performance. This helps sequential performance. After I read sector 1, I have half a disk rotation (at 300rpm or 5/s that's about 10ms) to queue up the read for sector 2. If I have a slow system, this can help performance a lot, since I don't have to wait for the disk to rotate a full rotation if I'm not fast enough to setup the transfer for second immediately. This is often impossible on slower systems since the inter sector gap is maybe 20-40 bytes. 40 bytes transfers at 250kHz, 320bits at 250kHz is about 100 micro seconds. At 4.8MHz that the Rainbow runs at, this is 5000 instructions. But since there's latency between when the transfer is complete to the controller, and the system interrupts this can easily have less than 10 microseconds to respond. On the Rainbow, it takes tens of microseconds to setup the command to read the floppy. Given such a tight time budget, you can get a big gain by requiring only 100ms to get the next sector instead of 200ms.

This is well known and well documented in the Rainbow community. I had to cope with it when I wrote IMPDRIVE back years ago by translating the requested sector so that it would read the proper sector when reading 3.5" floppies which don't do this.

There's another way to get better performance. MS-DOS and CP/M don't do this. But Venix does. If you want to reduce the amount of time that you have to wait for sector 1 to spin by the head when stepping from the previous track, you can skew the sectors another way. If you shift the start of the interleave sequence above to be something like 8, 4, 9, 5, 10, 1, 6, 2, 7, 3. This gives you half a rotation to get the head moved and settled. If sector 1 is right under the head by the time the head is done moving and the host gets its command to the floppy controller, we don't have to wait an average of half a rotation.

So to sort out this, I dd'd one of the disks under Venix ono the disk. Tar was able to read it readily there, but couldn't on raw images. So, I wrote a program to undo these effects. Once I did that, I was able to read the images with tar.

Once I was able to decode the images, I discovered that the floppies I'd read out under dos using rbimg. icreate.exe is used to read it out, and iwrite.exe can be used to recreate them. These days, this is the gold standard to preserve disks. Well, non-copy protected disks. If you want to preserve the disk in all its glory, use a disk imaging solution like Kryoflux.

Not lining up

Close, but not quite

So, it looks like I'm getting closer. I read the disks with the 'physical' read option, which did sector skewing. Now that I've looked at them, I think that was a mistake. It looks like they are meant to be read in a logical mode, so I'm redoing a few to analyze again.

So I wrote a program to try to sort that out. It did work, but not quite. It got things almost right, but over-shot the mark when it came to getting the second or third file from the tar ball. It looked for all the world like there were missing sectors or something.

On a lark, I decided to try to read 11 sectors instead of 10 that RX-50's are usually formatted at. And I was able to read all 11, and the data for the 11th doesn't seem to match anything else. But we still have a mismatch, and I'm trying to chase that down. Well, 11th sector for tracks 2 and larger. So, I need to try to figure out why I'm not seeing extra data between the files.

So it looks like I'll have to do this again to get all the bits.... I think I need to puzzle this out with some friendly files that have text (include files) in them. maybe there's 12 sectors, though that would be unprecedented for these drives. You can do 800k like the RX-50, and also 880k for the amiga, but I don't think you could fit much more than that on a track, but who knows, maybe there is 12 sectors and I'm messing up.

Then again maybe I should just wait for the Kryoflux to show up so I can image the entire disk, including the boot disk.

Peering into the Venix Disks with read errors

As you may recall from last time, I had 5 errors on two disks I'm trying to read for the Rainbow Venix Floppy project.

The specific errors were on two disks. 76/3, 77/7 and 78/5 on the "User 3" disk and 78/5 and 79/5 on "User 2". The program I was using to read the disks didn't report the kind of error, so I'll have to augment it to do so.

This is very concerning to me, so I though I'd take a closer look what's on the disks. It appears that the disks are much simpler than I'd though. The first two tracks (10k) is nothing at all. Then we have what appears to be a TAR file after that. So, it looks like, worst case, we have at most 5 files that are corrupted, and more likely only a couple. The distance on User 2 is only 5k, while the distance on User 3 is 11k. There's a good chance that there will just be one file that's affected.

If I wanted to try to extract these disks, there's another wrinkle. I thought a simple dd to remove the first 10k would do the trick. However, there's a snag. I read these in the physical mode. The RX-50s, at least as used on the Rainbow, have a sector interleaving starting with track 2. The actual sectors are labeled as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 on the disk (that's the order I read them out in). However, it appears that these sectors correspond to 1, 6, 2, 7, 3, 8, 4, 9, 5, 10, so I'll have to write a stupid filter to de-interlace them. Otherwise TAR throws all kinds of fits.

Unix v7 Tar files, btw, have fixed 512-byte headers. Wasteful, to be sure, but great for data recovery. A quick grep from hexdump shows what's on the 'User 1' disk:

00001400  2f 75 73 72 2f 61 64 6d  2f 00 00 00 00 00 00 00  |/usr/adm/.......|

00001800  2f 75 73 72 2f 62 69 6e  2f 00 00 00 00 00 00 00  |/usr/bin/.......|

00001c00  2f 75 73 72 2f 62 69 6e  2f 61 64 62 00 00 00 00  |/usr/bin/adb....|

00009200  2f 75 73 72 2f 62 69 6e  2f 72 61 6e 6c 69 62 00  |/usr/bin/ranlib.|

0000c600  2f 75 73 72 2f 62 69 6e  2f 63 61 6c 00 00 00 00  |/usr/bin/cal....|

0000e000  2f 75 73 72 2f 62 69 6e  2f 61 63 00 00 00 00 00  |/usr/bin/ac.....|

00010800  2f 75 73 72 2f 62 69 6e  2f 62 63 00 00 00 00 00  |/usr/bin/bc.....|

00011400  2f 75 73 72 2f 62 69 6e  2f 64 6f 73 63 6f 70 79  |/usr/bin/doscopy|

00014400  2f 75 73 72 2f 62 69 6e  2f 63 62 00 00 00 00 00  |/usr/bin/cb.....|

00014600  2f 75 73 72 2f 62 69 6e  2f 64 63 00 64 63 00 2d  |/usr/bin/dc.dc.-|

00015000  2f 75 73 72 2f 62 69 6e  2f 63 61 6c 65 6e 64 61  |/usr/bin/calenda|

00016c00  2f 75 73 72 2f 62 69 6e  2f 63 6f 6c 00 00 00 00  |/usr/bin/col....|

0001a200  2f 75 73 72 2f 62 69 6e  2f 64 6f 73 00 00 00 00  |/usr/bin/dos....|

00020400  2f 75 73 72 2f 62 69 6e  2f 64 65 72 6f 66 66 00  |/usr/bin/deroff.|

00021e00  2f 75 73 72 2f 62 69 6e  2f 65 67 72 65 70 00 00  |/usr/bin/egrep..|

00026a00  2f 75 73 72 2f 62 69 6e  2f 66 67 72 65 70 00 00  |/usr/bin/fgrep..|

00027a00  2f 75 73 72 2f 62 69 6e  2f 67 72 61 70 68 00 00  |/usr/bin/graph..|

0002be00  2f 75 73 72 2f 62 69 6e  2f 69 70 6c 6f 74 00 00  |/usr/bin/iplot..|

00031a00  2f 75 73 72 2f 62 69 6e  2f 6c 65 78 00 00 00 00  |/usr/bin/lex....|

00039e00  2f 75 73 72 2f 62 69 6e  2f 73 70 6c 69 6e 65 00  |/usr/bin/spline.|

0003c400  2f 75 73 72 2f 62 69 6e  2f 6c 70 73 74 6f 70 00  |/usr/bin/lpstop.|

0003f800  2f 75 73 72 2f 62 69 6e  2f 6d 34 00 00 00 00 00  |/usr/bin/m4.....|

00042a00  2f 75 73 72 2f 62 69 6e  2f 6e 65 71 6e 00 00 00  |/usr/bin/neqn...|

0004a600  2f 75 73 72 2f 62 69 6e  2f 6e 72 6f 66 66 00 00  |/usr/bin/nroff..|

00054600  2f 75 73 72 2f 62 69 6e  2f 73 70 65 6c 6c 00 00  |/usr/bin/spell..|

00054c00  2f 75 73 72 2f 6c 69 62  2f 74 6d 61 63 2f 74 6d  |/usr/lib/tmac/tm|

00055600  2f 75 73 72 2f 62 69 6e  2f 79 61 63 63 00 00 00  |/usr/bin/yacc...|

so there's lex, yacc, and several familiar filters. Transferring from the Rainbow is being super slow since I'm sending them via old-reliable KERMIT, so I've only looked at the first file.

All in all, I'd say this is encouraging news. I'll transfer the other disks and see what's missing to see if it is worth trying to delve further into the missing bits or not. If nothing else, it means we can use whatever data we can for those 5 sectors and the reduced functionality may be almost nothing. And there's a chance we can get a second copy of the affected files from the boot disk or something.

Rainbow 100 Venix/86R Disks Found

Recently, there was a post on an obscure retro-computing blog. It said that someone had acquired a Rainbow 100, which isn't so unusual (though there's only a few on the market). The part that was interesting was that it said it included VENIX disks. After some email back and forth, the owner sent the disks to me for reading.

After fighting for a week with PCs (more on that in another blog), I decided to search to see if there was a Rainbow native solution. Turns out that there was something called RBIMG by PrintStar (Jeff Armstrong) which I was able to get going on my old Rainbow 100B. It came with source because the Venix disks were mostly good, but some sectors requires a number of retries. It appears to be a Version 7 Unix for the PC.

I've read in the Boot disk, a bunch of Xfer Disks (not sure what they are), All the User and System utilities and an BSW extension disks. That's the good news. The bad news is that I still have a few sectors that aren't doing so hot. User3 disk has 3 errors, User2 disk has 2 errors and the boot disk has 1 error.

The Boot disk's one error, though, seems intentional. I've actually booted the Venix disk on my Rainbow, and it comes up. It also tells me the serial number after what sounds like a read error. I'm not yet sure what scheme that Venix uses, so that one might be solvable. It's cool to see a unix kernel that's only 45k in size.

So I'm 5 errors away from having a full set of disks. My thought was that maybe these disks were produced with a 1.2MB drive instead of a real RX-50, so diversity being good, I'm going to try to bring up an old PC I have sitting on the E-Waste pile. I have newer ones, but the BIOS doesn't support 1.2MB drives in those, which may be a clue that 1.2MB drives won't work at all in them (the literature on the web is spotty on this topic, but seems to indicate that latter-day systems cost-reduced 360k/1.2MB support out of the controller). More on those trials and tribulations in a newer post.

Installing FreeBSD-current with 11.0R installation image

Just a quick blog to document a trick.

Boot the install image (or even just a boot-only image). Get a Shell. At the shell, type

# env UNAME_r=12.0-CURRENT bsdinstall auto

and it will get the latest 12.0-current build installed. Thanks to Allan Jude for alerting me of this trick.

Making Cables

Tonight I assembled my cables I'll need to read my Rainbow 100B ST-251-1 hard drive I've had for a long time to get a backup.

I recently purchased a MFM ST-506 emulator which supports reading and writing old hard disks. To read the ST-252-1 I need the old standard two-cable (20-pin and 34-ping) to connect it to the emulator. This will give me a final backup, and also make it easy for me to snag the data from the drive.

Once I get this done, I'll switch to using the emulator to access the drive so I'll have continuous backups. It will also make it possible for me to explore Venix if the Venix disks that I just got are still good.... Still need to read them. I've hit a few snags in reading them from FreeBSD, which appears to have broken ISA DMA in 11.x, so I'll have to install 10.x instead of -current.

We'll see how it goes...

Compiling sdcc for FreeBSD

In preparation for a possible new project, I needed to build sdcc, a compiler that targets a bunch of 8-bit architectures. You can find information about sdcc at their sourceforge page: http://sdcc.sourceforge.net/

Usually, this is just a port away. But due to looking at the wrong tree, I thought I had to compile it myself. The port and package are great, and are what people should normally be using, but the problems showed the need to use a trick or three to get the job done. I thought I'd document them here, since they are minimal and easy to write up.

FreeBSD's compiler doesn't default to having /usr/local/include in its include path, or /usr/local/lib in its library path. Normally, one would try to get around this by just add --prefix /usr/local to the configure command. However, that doesn't work for the sdcc stuff (since it is already the default). You have to explicitly add more things to the command line. After some digging, it turns out it's only a few variables to worry about:

./configure CC=clang CXX=clang++ CPPFLAGS="-I/usr/local/include -L/usr/local/lib" CFLAGS="-I/usr/local/include -L/usr/local/lib"

 The key ones here are CPPFLAGS and CFLAGS.

With that, I have a build sdcc. But it's a pain to install. The port / package is much easier for that reason alone.

Cool new FreeBSD 11/12 kernel developer trick

One of the cool new features in FreeBSD 11 is that we can use the system compiler as a cross compiler, eliminating the need to rebuild clang. While this allows buildworld to go much faster, there's another benefit that might not be immediately obvious.

For kernel development, this almost means we can forget about the usually required 'make kernel-toolchain' even when we're cross building. The system linker can't (yet) handle it, so the kernel fails to link if you've not done it. The good news is that you don't need to build gcc or clang for the kernel toolchain target, so you can build it quickly (in under a minute on my fast build machine, under 5 on my laptop VM). The trick is one command:

make kernel-toolchain TARGET=arm WITHOUT_{GCC,CLANG}{,_BOOTSTRAP}=t

If you are developing for amv6/v7, change TARGET=arm to TARGET_ARCH=armv6. I've not tried others, but suspect they will work as well as clang supports the target in question. This allows me to start building test kernels w/o the usual delay of building a kernel toolchain, especially on my laptop VM where  I'm chronically short on disk and always deleting the old obj tree.

This works on both 11.0 and 12.0-current. It likely will fail on 10.x since the system compiler as the cross compiler work hasn't been MFC'd there yet.