FreeBSD Lua Loader (started in GSoC 2014) about to land

Just a quick note. I've been working on making the /stand environment easier to deal with and integrate into. After months of work on and off, the using Lua as a boot loader extension language is ready to roll into -current.

A preview review can be found here.


Setting up a SVN mirror

Setting up the svn mirror

Just following the steps in
in the Setting up a svn mirror (5.4.7) but have found there's some issues...

First step, grab the svn seed file
% fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r290116.tar.xz
this takes about 30 minutes. Also grabbed svnmirror-ports-r400410.tar.xz for a ports mirror. These file names are different than the instructions.

Next extract this somewhere (say /home/svnmirror/base) and then update it with "svnsync sync file:///home/svnmirror/base" and then set it to updating. You can do similar things for docs and ports. These files are a couple of years old now, so the sync still takes a several of hours. I believe that this is due to the protocol round trips required to extract it being sensitive to the latency between my machines and the FreeBSD svn server.

I use the following shell script as a cron job to keep things up to date.
lock="lockf -s -t 0 -k /tmp/svn-sync-mutex"

${lock} svnsync sync ${URL}/base >> ${logdir}/svn-base.log 2>&1
${lock} svnsync sync ${URL}/ports >> ${logdir}/svn-ports.log 2>&1
${lock} newsyslog -r -f ${logdir}/newsyslog.conf


FreeBSD cumulative commit graphs

FreeBSD Commit Graph

I've updated an old graph from a talk I gave in 2010 to have data up through today. The rate of development seems fairly steady.

When last I did this release, stable branches were flatter and lived longer. Now there's a fair amount of activity after the branch that continues for years.

Note: The hash marks are generally releases. I've removed the 'artificial' commits that were needed as a result of the CVS to SVN migration the project did a few years ago. The 2.0.5 branch has been omitted, and the 2.2.9 release (never tagged in the repo and < 10 changes different than 2.2.8). I had release numbers on the hash marks in earlier revs, but that's too busy now.


Reading in my Rainbow Floppy Collection

Recently, I acquired a kryoflux board to help read in the Venix floppies... I finally have it up and running.

I've managed to read in most of my collection of ~300 Rainbow floppies. 95% of them were actual Rainbow disks. The other 5% were either completely blank, or some variety of IBM format (including 3.5" 720k on a 5.25" floppy!) with a couple of 'robin' formatted ones just to keep you on your toes. I didn't wind up seeing any of the very rate dual-sided RX-50's (they exist, but are super rare because this was a home-brew job not an official thing).

While it is possible just to do
dtc -ffoo.img -i4
It takes a long time since it's reading the back-side that has no data on it. Using
dtc -ffoo.img -g0 -i4
is all that's needed to go fast. Some caveats should be observed.

Nearly every track has extra data in at least one of the sectors. This threw me for a loop, but is harmless (as stated in the manual).

I had some issues with old, ill-maintained floppy drives. Bought a refurbished one and all my troubles just went away. I had several old drives and based on the good floppy drive, I could find known good floppy diskettes and was able to refurbish one of my old units. I had the best luck with the TEAC FD55-GFR drives, btw. Not sure it's worth while to refurb the rest of the units, though, since I have something that's working...

Keep an eye on the number of sectors reported. It's usually 10. If it isn't, stop right away. It isn't a Rainbow / RX-50 formatted disk. If it's 8 or 9 sectors, it's likely an 320k or 360k IBM floppy. Removing -g0 and adding -k2 is useful for these diskettes (saves time more than anything. If it's 15 sectors, it's likely a 1.2MB floppy. Had 2 or 3 of those mixed in with these diskettes despite never having a machine that produced them.

As these disks age, you may need to try multiple times. I have some disks, though (maybe 1%) that I can't read back certain tracks on even with that. The default is good, but trying more helps with a similar number (so -t10 -tc5). I don't know if there's a way to retry individual sectors, or if dtc is internally buffering good sectors from previous retires or not. Previous programs I've had on the Rainbow for extraction had great luck retrying sectors multiple times when the full track read didn't work...

Some disks are copy protected. Not sure how best to preserve these disks. The -i4 format won't work with, for example, Lotus 1-2-3 system disks because tracks 78 and 79 have sectors numbered up to 12 and are missing sectors (they do this with a special format that I believe omits sectors 9 and 10 and instead has sectors 11 and 12). There are other copy protected disks that have deleted sectors since the Rainbow could easily read those, but they were hard to create from standard interfaces. Don't know if -i2 images would help, or if I have to go all the way down to -i0 to ensure proper imaging. The copy protected disks, however, were all broken in the '80s, so preserving them isn't so interesting to me. There's also a couple of disks that have data on track 82, but it's marked as track 79 and it's unclear what this data is. I know there are several old formats that preserve this info (TELDISK and DISKIMAGE being two that I've run into most).

cpmtools is able to extract data from CP/M formatted disks (use dec_pro format). It understands all about the interleaving you need to do on tracks 2+ and can work on raw kyroflux images.
As for MS-DOS disks, I wrote a small utility to convert the physical layout to a logical one, and to (optionally) replace sector 0 with a standard boot record with the BPB in it (Rainbow floppies have Z80 boot code which has a different starting byte (f3) and doesn't have the tables expected. Doing this, both mtools and FreeBSD's msdosfs can copy files easily enough. But so far it's not reversible, so you can't (yet) use it to create floppy images. I'll be testing two tricks (just prepending the magic sector and appending sector 0 at the end) to see if that might help. I can post a pointer if there's interest.

Venix disks are a mixed bag as far as data extraction, but are otherwise boring: 80 tracks, 1 side, 10 sectors per track. Just had brief access to the distribution diskettes, mostly prior to having kryoflux (and the reason for buying it). Ironically, I couldn't get the kryoflux working due to the drive issues I discussed above before I had to return the distribution diskettes. But I was able to copy them natively on my Rainbow...

Finally, there's still 2 or 3 disks that I've been able to read in that I have no clue what the format is. They are kinda sorta CP/M, but also kinda-sorta MS-DOS from looking at the first two tracks, but neither quite works to decode them. Physically, they RX-50. They don't look like ODS-2 nor v7 unix file systems, but that's harder to know for sure. They are unbootable (tracks 0 and 1 are 0xe5, the erasure pattern for RX-50s).

My only real complaint is that it would be nice if dtc would spit out a summary at the end, including tracks unreadable. I have to keep an eye on the logs otherwise. Not a huge deal, to be sure.


v7 compilation for simple .c's in /usr/src/cmd.

After about 4 and a half hours of build time, I have results to share. There's 110 .c files in /usr/src/cmd. 85 of these compile and link (it's unclear if they run). 25 have a compile or link error:

  • accton, sa (lack of acct in libc)
  • ar, nm, prof, ranlib, size, strip (mismatch between NMAGIC/OMAGIC and ARMAGIC_X)
  • arcv (obsolete conversion program written in old style)
  • clri, dcheck, dump, dumpdir, icheck, mkfs, ncheck, quot, restor (inode differences)
  • dmesg (message buffer issues?)
  • getty (freaks out compiler)
  • graph (missing symbols for curses and libm stuff)
  • osh (old shell? Written in funky style)
  • ps, pstat (proc definition differences)
  • tc (silly compiler issue, trivial to fix, but I have not Tek4015 terminals)
So 10 classes of issues for the affected files. Not too terrible. But there's other issues that will need looking into. 'ld' compiles fine, but it works on pdp11 a.out files, for example. cc compiles, but doesn't have the venix specific switches. I've written a shell script to do this, not a Makefile, and the commands used aren't quite right. Venix appears to support different amounts of stack under the data segment (but without docs, it's hard to know what flags I need), but none of my binaries have those. There's NMAGIC and OMAGIC binaries that have different meanings, and so on. All this makes the 'file foo /bin/foo' differ. Plus I haven't done the more complicated commands, nor have I poked at the compiler apart from cc, nor the kernel really at all, nor the libraries, nor the .y files. And factor.s and prime.s are pdp-11 assembler, so I'd have to pull those in from somewhere.

But given that almost all the normal commands compiled just fine, and many appear to work gives me hope that I might be able to reconstruct the sources used to build at last parts of the system. For some reason, that's appealing to me, but I can't really articulate a reason why, or come up with why that might be useful. Though it seems like a fun background project to celebrate the 40th anniversary of Version 7 in a couple of  years. Though running it on a 35 year old computer at that point may limit its appeal...

Rebuilding v7 sources on Venix, early results

I just started rebuilding the v7 sources on my Venix Rainbow on a lark. I'll post a full writeup, but here's a two early things I've learned.

First, it takes 1 minute to compile cat.c into cat. Yes, one full minute. On my normal server, it takes .1s of elapsed time. This is a 600x speedup since then, likely more since clang isn't known for being gentle on system resources.

Second, some compiler error messages are classic:
"getty.c", line 12: compiler error: insane structure member list
I have always thought getty to be rather opaque to use. The code is somewhat simple, but that simplicity doesn't lead to easy understanding in this case.

Most things not dealing with the filesystem are building w/o a problem. This is a pleasant surprise. Though the speed at which they are building isn't going to set any records. About 5% have issues that prevent compilation or linkage out of the box, though the problems look easy to fix. I started the build at 9:30. It's now 2 hours later and we're to building 'grep'. We're on pace for a rebuild of just the trivial foo.c -> foo programs taking 5-6 hours. This isn't for the more complicated programs like make or tar that have their own subdirectory.


Rainbow Venix with BSW Enhancements

I've taken some time to look at the Venix disks. I've managed to read them all after retrying the read errors enough. I've created new disks and installed it on my system. I'll go through all that in a forthcoming post since there's lots of fiddly bits.

This post I'll expose what BSW is. It's the Boston Software Works. It was a company that was run by Larry Campbell from 1985 until 1995. They had a number of different products over the years. Through about 1988 they offered their enhanced version of Venix for $800 a copy (not their choice on the price).  It was normal Venix, only better. Larry posted the following to mod.newprod on Usenet back in December1986 describing what it was. I've edited the formatting a bit, and trimmed the headers:
The Boston Software Works (BSW) is now distributing an improved version of VENIX/Rainbow.  VENIX/Rainbow is a complete UNIX(tm) system for the DEC Rainbow 100, developed by VenturCom, Inc. of Cambridge, Mass. It is a V7-based system, with extensions such as plot(3) drivers for the Rainbow color graphics board and DEC LA50 printer, semaphores, and shared memory.
Standard VENIX/Rainbow requires and supports the DEC RD51 winchester disk (10MB) only.
The Boston Software Works enhancements to VENIX/Rainbow include:
  1. Support for any disk physically connectible to the Rainbow. Our development system runs a 70MB Micropolis disk.
  2. Virtual consoles, as found in VENIX on IBM-compatible machines, and also in XENIX and Microport System V/AT.
  3. Support for the NEC V20 chip, which replaces the Intel 8088 for a 10-15% performance gain.
  4. Support for simultaneous use of a monochrome display (for terminal/console use) and a color graphics display (for graphics).
  5. Ports (on an as-is basis) of several public domain programs, such as the Usenet news software, rn, smail, and Jove, an excellent EMACS subset.
  6. Support for the CHS dual-winchester controller.
The price (single copy) for VENIX/Rainbow is $795 and includes a complete manual set.
I've installed both the original Venix/Rainbow and this enhanced version. The enhanced version is the way to go. It knows how to cope with the larger disk I installed in my Rainbow, and the installation process was much less painful (though there were some up-front gotchas that I wound up having to re-run WUTIL to get through).
These are all cool things. The only issue that I've seen is that the .o's aren't included so it's impossible to rebuild the kernel with these neat, new features without significant hassle. The other problem is that while I can get stable file transfer on MS-DOS with LCTERM at 9600 baud, I can't with Venix. 4800 is error prone even. Messing with the serial chip seems to also trigger the dreaded Interrupts Off message, so I've not messed with it too much. Still don't know if that's my Rainbow going bad, or other issues (have some spare parts arriving soonish). The maximum throughput is only 240-300 CPS anyway with LCTERM, so maybe I'll just run at 2400 or 3600 baud.
Not listed above: it kinda seems like it is reading my clikclok. I did have to fix the 'date' command to grok Y2k though (well, since I don't have the sources, I used the Unix v7 sources from TUHS). If this keeps up, I'll put together a distribution of useful things from V7 :).
I'll blog more when I've run through another install and worked out how to boot in MESS (it has a bug when booting directly off the hard drive, but maybe the BOOT program will fix that).

[[ Edited May 8, 2017 to supply link to original mod.newprod article on google groups ]]


Rainbow 100 Windows 1.0 disks, redux

Rainbow 100 Windows 1.0 Disks

OK. I never thought I'd find myself writing a blog about Microsoft Windows of all things. However, that's what I'm doing.

A while ago, on the cctalk mailing list I mentioned that I had Windows 1.0 disks for the Rainbow. I didn't think anything of it until someone contacted me about it. I'd never looked at them in the 15 years they've been in my collection. You needed the Graphics Option card, and I don't have one so I just filed it away in the back of my brain. I never looked into the situation further.

So, it turns out that the publicly available Rainbow Windows 1.0 disks are really not quite right. They are a collision between the Rainbow 100 version, and the VAXmate version. And evidentially, neither quite work. This collection has been cargo culted around in the three decades since they became available, but they didn't work. it was well known, but the archives contained what they contained and successive generations of hackers tried, and failed, to disentwingle the resulting mess. The mess is still available at archives of archives of archives, for example at Classic CMP there's a copy of the "goodnight" archive which is an incomplete copy of the various BBS archives that existed in the last 80's and early 90's.

Sometime in 2001, Arun Welsh and I corresponded about the DEC Rainbow. I forget the details, but if memory serves, he had a ton of disks that he was looking to get rid of and wanted to see them find a good home. The details are frankly fuzzy after 16 years. I got these disks and took out a few of the interesting bits, but mostly it was a collection of files that were available elsewhere so I didn't look too closely at the collection. But DEC Windows 1.03 stuck in my brain for some reason.

Recently, that boast I made in cctalk generated some email requesting a copy. Since I had my Rainbow out and was messing with it, I thought I'd oblige them. Once upon a time, I could read Rainbow disks, but that was a dozen hardware iterations ago, so I fired up my old machine and read the files in. There's two sets. One was Rainbow MS Windows and was 7 disks. It contained what appears to be a copy of the Goodnight archives with the entertwingled set of files. The other was 4 disks of DEC Windows 1.03.

The other one proved more interesting. It was a different set of files. It came with a letter tucked into the one of the disks giving permission to upload the set of disks to any BBS (including the recipient's BBS, who wasn't Arun, I've forgotten his connection to this mystery). If there's interest, I can post this letter.

Early repots suggest this is a good copy. If they prove true, I wonder how hard it would be to get things corrected in all the online archives.

I've uploaded to a github repo I've created. These files are under 'decwin' while the putative copy of the goodnight archives is under 'rbwin'. I've also uploaded a copy of wutil 3.2 source and binaries. I plan on uploading the Venix binaries there as well, once I get things sorted out and post instructions on how to install them either on real hardware, or in an emulator.