2.11BSD Missing Patches

While looking into some date anomalies in the final image (since I'd like to get the dates right) I discovered a number of source directories had dates slightly newer than the date in the announcement. This lead me to discover some missing patches in a couple of different places.

The Anomaly

I've automated the system generation, tape generation and installing from tapes to allow me to make small tweaks and get end to end testing. As part of this, after the system is installed, I'll do a test boot, similar to the following, as if I'd installed the system on April 10th, 1991 and booted it on April 15th. The boot looks something like this:

sim> boot rq

boot: 73Boot
: ra(0,0)unix

2.11 BSD UNIX #1: Fri Mar 15 15:48:55 PST 1991
root@wlonex.imsd.contel.com:/usr/src/sys/GENERIC

phys mem = 4186112
avail mem = 4008640
user mem = 307200

Apr 10 13:50:01 init: configure system
ra 0 csr 172150 vector 154 attached
rl 0 csr 174400 vector 160 attached
tms 0 csr 174500 vector 260 attached
ts 0 csr 172520 vector 224 attached
xp 0 csr 176700 vector 254 attached
erase, kill ^U, intr ^C
# date 9104151234
date: can't write wtmp file.
Mon Apr 15 12:34:00 PDT 1991
# Fast boot ... skipping disk checks
/dev/ra0c on /usr: Device busy
checking quotas: done.
Assuming non-networking system ...
preserving editor files
clearing /tmp
standard daemons: update cron accounting.
starting lpd
starting local daemons:.
Mon Apr 15 12:34:01 PDT 1991

2.10 BSD UNIX (my.domain.name) (console)

login:

I set the date in single user then bring it up to multiuser. In one of my tests, I found the following:

-rw-r--r-- 1 imp imp 9777 Aug 31 1991 alloc.c
-rw-r--r-- 1 imp imp 4817 Aug 31 1991 alloc11.c
-rw-r--r-- 1 imp imp 12474 Aug 31 1991 doprnt.c
-rw-r--r-- 1 imp imp 3299 Feb 23 1987 doprnt11.s
-rw-r--r-- 1 imp imp 831 Aug 31 1991 printf.c
-rw-r--r-- 1 imp imp 20446 Aug 31 1991 sh.c
-rw-r--r-- 1 imp imp 1771 Aug 31 1991 sh.char.c

which I thought was quite strange. There shouldn't be any files dated newer than the release in the tree. I know all files patched don't get the time set right, so I exclude those from my search (I plan on fixing that bug later). The above files (and others, it's just a short list) shouldn't be there. So I started looking...

The Diffs

Running diffs against 2.10.1 I discovered that the csh files were almost all the same. However, a typical diff looked like:

diff -ur root-2.10.1/usr/src/bin/csh/alloc.c root-2.11/usr/src/bin/csh/alloc.c

--- root-2.10.1/usr/src/bin/csh/alloc.c 1987-02-08 15:27:23.000000000 -0700

+++ root-2.11/usr/src/bin/csh/alloc.c 1991-08-31 01:03:00.000000000 -0600

@@ -4,10 +4,10 @@

* specifies the terms and conditions for redistribution.

-#ifndef lint

+#if !defined(lint) && defined(DOSCCS)

/* From "@(#)malloc.c 5.5 (Berkeley) 2/25/86"; */

static char *sccsid = "@(#)alloc.c 5.3 (Berkeley) 3/29/86";

-#endif not lint

+#endif

* malloc.c (Caltech) 2/21/82

which removed the SCCS IDs from the binary to save size. Other changes included introducing overlays for the first time. This indicated size issue. Let's take a look at what else was going on around the 31 Aug 91 in the patch stream. Looking, we find that this is just after patch 18 (which fixed a long vs int bug in test) as well as patch 17, which updated pcc. This sounds like a size hack by someone that had just updated the compiler, or was testing with pcc (the normal system compiler wasn't pcc, but the earlier Thompson compiler). Another of the changes also fixes an issue with character handling, which other patches have done to reduce the size of binaries that got too big.

So, in context, this change makes perfect sense. The only trouble is that it wasn't posted to comp.bugs.2bsd, nor did it make it into Steven Schultz's patch repo. And csh isn't the only troublesome one. There's issues in rn, and games/warp.

The Catch Up Patch

There was a catchup kit that was issued officially as Patch 80 (though it omitted patch 79). Looking in that patch kit we find this change! So it's in one that was intended. So what to do? And it turns out there's 5 such patches (but only 4 of them made it into the patch kit... I'll talk about #5 in a minute).

I've decided to look at the dates of each of these patches and pretend they happened just after the patch whose date is closest). I've updated my mk211bsd script to extract these from the catch up patch.

Oh, and there were a number of new programs added in the catchup patch as well. These must be deleted too, but I'd already noticed that and deleted them.

tftp changes

So, on May 4th, 1991 a patch to dd.c was posted to comp.bugs.2bsd. It's also included in the official archive as patch 1. The release announcement was dated March 14th, 1991. But there's tftp files dated May 15th, 1991. What's up with those? Turns out, this is another missed patch (but one that's assumed to be in place in the catch-up patch because it's not in there. Well, it's partially in the patches, partially in the scripts. It's an update of tftp and tftpd to a new version. It was posted to comp.bugs.2bsd on May 15th, 1991, but isn't in the official list of patches. So not only do we have to dig it out of the catch-up patches (from two different files), we also have to restore the old man pages from 2.10.1BSD, but in a different place, so this patch will be a patch + rm (which going backwards is patch + cp)

csh changes

As discussed above, these are various hacks to get the size of csh down.

warp changes

The changes here are around the config script used to generate files for the build. The changes use full path names, and cope with the new shadow password format changes.

pcc changes

[[ edit -- these were poorly documented: they are in patch 17, but not called out specifically to apply ]]

rn changes

As part of the catchup, there's a number of minor patches to rn that were included in the catch up, but weren't formally published or occupy a number of their own.

My Work

So, how does this affect me? Well, it means that I need to understand the catch up patch a lot better. I had hoped to use it later as a cross-check against my work. I didn't anticipate that I'd be using it "sooner" as to get missing bits I'd found using other techniques. I've had to update mk211bsd to extract the bits, as well as creating a couple of hints files to help me undo the changes.

And when the time comes to reply all the patches, I'll need to take these anomalies into account as well. But that's a problem for future me.

I also have to complete the audit of the weird file dates. There's 63 of them right now, 29 in /usr/src still. Some of clearly old man pages that I can remove. Some are the result of running 'configure' or similar script (rn has 7 of these). Some are config files that change over time (like for the root name server). Some may be just left-over detritus of a running system. I need to see which ones fall into which categories and update accordingly. This may dovetail back into needing to bring them back to make sure I can march back to pl195 and get the same system. Since I started with ~1500 such anomalies, I think being down to 63 is quite good. And there's others elsewhere in the system...

Current status

As you might guess, if I'm finding things like this, that means I'm getting closer. I've shared a lot of this on my @bsdimp twitter account, but now is a good time for a wrap up. Here's what's done currently:

Script to undo all the patches, including helper 'hints' scripts, where possible from existing artifacts.
Miss patches reconstructed and integrated into the build
Automated installing 2.11BSDpl195 image
Automated bootstrapping back from 195 -> 0. There's a number of interesting problems here that I'll blog about soon
Building the 2.11BSD pl 0 tapes automatically
Test installing the pl 0 system from the pl 0 tapes.

The missing bits include

Getting the dates right (or failing that plausible) for the patched files
Finishing the date audit and tracking all anomalies to ground.
Cleaning up my helper scripts off the image
Creating a github repo with all the patches in it
Reproducing the build on a second system
Getting the ownership right for some files (eg using the mtree hack to get the ownership and permissions right, generating it from the pl195 tape/image, etc)
Getting dates right on /, right now restor(8) doesn't restore the date in one at a time mode, so these are all wrong.
Fixing tmscp boot. It's broken. The tmscpboot.s, ported from tkboot.s, only existed for a short period of time and has been lost. My reconstruction has issues (it won't boot), and I've not delved into why.
Creating automation to ensure that the 'catch up' kit will apply cleanly.

And of course, I need to figure out the best way to publish the artifacts when I think I'm done.

Warner's Random Hacking Blog

20200803

Missing 2.11BSD patches