Traditional Unix Toolchains

Older Unix systems tend to be fairly uniform in how they handle the so-called 'toolchain' for creating binaries. This blog will give a quick overview of the toolchain pipeline for Unix systems that follow the V7 tradition (which evolved along with Unix, a topic for a separate blog maybe).

Unix is a pipeline based system, either physically or logically. One program takes input, process the data and produces output. The input and output have some interface they obey, usually text-based. The Unix toolchain is no different.

Overview

Here's a simplified view of what's going on. We'll add more detail later.

In this view, the C compiler takes .c code and turns it into assembler. How it does that, and how it optimizes, etc is for another blog post. Once the assembler is created, it's passed to as(1) which translates the assembler into .o files. The .o files contain the binary representation fo the assembler, plus a lot of metadata about it: what addresses correspond to what symbols, how to relocate the raw assembler when connected together, various debugging information (sometimes) and what section each bit of data resides in. You cannot directly execute a .o file. ld takes all the .o files and produces an executable (the default name of which is a.out). a.out files are executable. They happen to be in the same format as the .o files, except they have a different magic number which tells the kernel how to load them into memory and initialize the CPU's registers for that program.

Program Layout

In traditional unix, there were only three sections to a program. There were no shared libraries or other fancy things done by the linker (such as linker sets). The world view was much simpler. There were three sections, each one had a size. There was the 'text' section. This was the executable code. There was the 'data' section, which contained initialized data. And there was the 'bss' section which also contained data that was initialized to 0. 

The heap resides above the bss and is managed by the unix sbrk(2) system call. The so-called "break" is set to the end of the bss segment (often referred to by the symbol ebss). Malloc(3) is built on top of sbrk(2) and will manage returning bits to the OS when it can.

For the PDP-11 and other segmented architectures, there can be complications. There can be separate I&D space (instruction and data) so that each one resides in it's own address space. This helps PDP-11 programs break the 64k limit. In addition, there can be overlays. Overlays are 8k segments that are mapped into the address space as needed to increase the text size of the program. The linker handles much of this, but the programmer must specify the overlap groups. Each group can be no more than 8k in size, and the main program can be no more than 56k in size (and the overlay manager uses 8k of the data segment as well). Programs in unix tend to not use overlays, but the kernel makes heavy use of them.

Compiler (cc, f77, etc)

Compilers create assembler output. Compilers, like the C compiler, may invoke other programs to do this. The C compiler runs the source through cpp to create an intermediate file (.i files) that it runs through the first pass of the C compiler. There are a number of other passes of the compiler that take the initial output and optimize it in various ways (usually by parsing and rewriting assembler). The final output of the compilers, at least in this era, is always textual assembler.

Assembler (as)

The assembler is the only thing in the system that creates .o files. Well, not strictly true since ld can also take .o files and produce a .o file from it, but true enough. The assemble takes the textual assembler and creates a .o file from that. The .o file includes information about how to relocate it (ld uses this info), what symbols go where, what bits are in the text section, what is in the data, etc.

Archive Files (ar)

It's fairly expensive to open a lot of files on Unix, especially if they are small. It's also inconvenient to carry around a number of different files to implement something. So unix ld also supports reading files in from an archive. An archive adds some headers to describe the file and then places a copy of the file into the archive. Unix has had a number of different archive formats (I could do a mini-blog entry on all of the ones through 4BSD), but conceptually they are all the same. The ar(1) program is used to create and manage .a files for this purpose.

One motivation for the library is to save space. If it's efficient to create a lot of tiny .o files, then the loader can only bring in what's needed saving space. When the address space is only 64k (or 64k+64k for separate I&D machines), every little bit helps. The archiver removes the overhead of having to do directory lookups on hundreds of files by creating a container for those objects that ld doesn't have to process by name.

When processing through a library, ld looks at each .o that's in the archive for symbols and includes it if it finds any that are needed for the current image so far. It just does one pass through an archive, though. This means if you have foo.o and bar.o in the archive and foo.o depends on bar.o somehow, foo.o needs to come first so that ld can find bar.o later. If they are in the other order, then ld may pass over bar.o entirely and then when it is processing foo.o, it will not go back and look for it.

lorder(1) and tsort(1) are used to try to optimize the order of the .o's in the library to make it possible to just do one pass through the library (though when you have circular dependencies, you're back to this same issue). lorder uses nm(1) to read all the .o's on the command line and produce dependencies. tsort takes these dependencies and sorts them into a list in the proper dependency order where possible. When cycles exist, it produces an order that minimizes passes required to resolve them all.

That sounds quite inconvenient, and it is. libc.a, especially in newer versions, has a number of circular dependencies that a single pass fails to resolve. One can work around this issue by specifying libc multiple times (which is unsatisfying, even if it doesn't produce a binary with two copies of everything), or do something else. The something else involves adding a table of contents to the library. A program called ranlib will read through an archive creating an index of defined symbols that points to the offset in the file that he .o with that symbol is present at. It does this by creating a first member of the archive named __.SYMDEF and placing its table in there. The format of the table is something that ranlib and ld agree on. ld uses this table to include files that are needed and to be able to seek backwards easily. In effect, this is the same as ld doing two passes over an archive (one to build this index, and one to process it), but in effect precomputes the first pass to keep ld simpler.

Loader (ld)

The linker can operate in two different modes. The first mode most people are not familiar with. In this mode, it will take .o files and partially link them together to produce a new .o file. libc uses this mode of operation to create all the assembler glue to call system calls and optimize out some of the local labels that the assembler produces and expects ld to optimize away. Since these files are consulted so often, the build process of libc does it at build time so that every invocation of ld later can be faster.

The second mode is the mode people are more familiar with. In this mode, ld combines a number of .o and .a files to create an executable (a.out by default, so that's what people call executables in general). a.out binaries lacked shared libraries, so ld produced the final output. This was both good and bad. It wasted space with all those copies of libc, but it also produced self-contained binaries that didn't need any external libraries to work. The PDP-11 didn't really have good demand paging hardware, like the later VAX machines, so was a poor fit to shared libraries. Shared libraries generally rely on mmap(2) working and larger address spaces to map the libraries in at. mmap(2) requires a page-grained MMU, which didn't usefully exist on the PDP-11 (it had 8k segments, which was far to large a percent of the whole address space to be useful). One good thing about binaries being self contained means that if the kernel can run the system calls that are in it, the binary will work. This has allowed PWB systems to be able to execute both v6 and v7 binaries, despite the two having different system call interfaces.

The Kernel

Speaking of the kernel, the kernel is the last step in the toolchain, or can be thought of as such. The kernel reads in the headers from the a.out files, and sets up the address space for the process when a new a.out binary is exec'd. It uses the layout I showed above, or some variant of it, to set things up, to populate memory and makes whatever arrangements with the MMU to protect the pages from other processes (if possible, some systems like an 8086 don't have MMUs but do have segments so can fake all this except the memory protection benefits). For separate I&D space binaries, it also sets up the segment registers for that to work.

The stack is also initialized. The detail of exactly where it goes varies somewhat. It's usually located with the data segment since it holds data almost exclusively. Stacks in this era were usually quite small. This tended to drive programs that had shallow call graphs and that made use of more global variables than a more modern style would suggest. All these things conserved stack space, though it's generally agreed today that it required more effort to read and understand because the context is spread out over more parts of the program than more modern coding practices tend to produce.

Conclusion

Without the complications of shared libraries, or link time optimizations, the tools of this area tended to be rather simple. They had simple interfaces between them. There were good boundaries between the different components. This limited the number of programs with knowledge of the formats for the different layers. Due to this limited spread of knowledge, switching out different parts for other parts often could be done without changing components that didn't directly know about the object format. This also produced simpler programs that used different engineering tricks to get the most performance out of the limited hardware of the day. The PDP-11s were approximately 0.1-0.5MIPS machines in this time frame with super slow I/O paths. This is about 100,000 times slower than most computers people interact with today. One advantage of the thoughtful engineering trade offs is that all the pieces are relatively easy to understand.

The modern ecosystems that we have today are more complex. ELF came along in the 90s and obsoleted the text, data, bss world view. shared libraries made huge programs, like X11, feasible. Today, clang bypasses the separate assembler stage and generates .o files directly. The llvm linker, lld, can optimize binaries between modules to produce better code. All these new features added complication to a simple model. While I morn for the loss of simplicity, I've become too used to the rich features they provide to want to go back. Understanding the roots of this complexity, though, helps to understand some of the weird quirks that persist, even to this day.

And speaking of weird quirks, I'd like to end with 'bss'. It's a 1950s IBM assembler mnemonic for 'block started by symbol' and was used to create storage that was associated with a symbol, but had no initial value. Today, 'bss' is no longer that, exactly. Its origin has been lost, for most people, in the sands of time and now it just means 'zeroed storage area'. So this very Unix centric term actually predates Unix by 10 or 15 years for a machine that Unix wouldn't run on until it was 10 or 15 years old... Here's a snapshot from the IBM assembler manual, available from the UA-SAP wikipedia page

showing the original source...

[[ This blog edited to include snapshot of the BSS manual entry ]]

2.11BSD Original Tapes Recreation

In Search of 2.11BSD, as released

Almost all of the BSD releases have been well preserved. If you want to find 1BSD, or 2BSD or 4.3-TAHOE BSD you can find them online with little fuss. However, if you search for 2.11BSD, you'll find it easily enough, but it won't be the original. You'll find either the latest patched version (2.11BSD pl 469), or one of the earlier popular version (pl 430 is popular). You can even find the RetroBSD project which used 2.11BSD as a starting point to create systems for tiny mips-based PIC controllers. You'll find every single patch that's been issued for the system.

What you will not find, however, is the original 2.11BSD release tapes. You won't find the original sources. With some digging, you can find is 2.11BSD pl 195. This was released about 30 months after the original was released, and is the oldest one that's known to exist. The reason is that the original 2.11BSD tapes were distributed by USENIX. They charged a large fee for the tapes, and so not too many people bought them. And this was before Caldera released the ancient Unixes under a permissive license, so the bulk of the feed went to AT&T. It's cost made it a low volume item. Plus, there were patches all the time, so the master tapes were respun from time to time. The originals weren't preserved, alas, because storage was expensive and by the early 1990s the PDP-11s were a bit of a fringe machine, except in certain niches with long procurement times...

But wait, you said we have all the patches, patch -R is super easy to use. Just use that to go backwards, right?

Well, no. The patches aren't all context diffs. Instead they include instructions like "remove these files, then extract this uuencoded compress tarball" or other information destroying instructions. So, the information is lost, maybe for good. We can't get there.

Or can we. If we look at it in a vacuum, it sure sounds hopeless. Information destroyed, you said. However, while it's true information is destroyed in many of the patches, it's only one copy that's destroyed. We have other sources of information. The 2.11BSD release is part of a series of releases in the 2BSD family, so we have 2.10.1BSD, the prior release. That's been preserved. We know from the release notes that significant influxes of code came from 4.3BSD. There's also a usenet news group called comp.bugs.2bsd that posted patches. It's known that these patches wound up in 2.11BSD (also all the patches to 2.11BSD were posted there by the original authors until usenet went away).

The Project

So, that brings us to my 2.11BSD pl 0 restoration project. The goal of the project is to create two main artifacts. First, it would be cool to have a git repo that has all the 2.11BSD patch points in it. Second, it would be really cool to have a near copy of the 2.11BSD release tapes. This project aims to create these artifacts in a reproducible way. When completed, anybody can take the existing artifacts we have, the scripts from the project (including all the hints needed to get the data from other projects, as well as a few hand-crafted patches which produce results consistent with all know info about these files).

Status

I've worked my way through the 195 patches undoing them. Many of them are simple patches, packed in an annoying eclectic number of different ways. Some, however, destroy information and require research to untangle. I've done the best I can and have made it back to patch level 0 sources (almost, there's one or two lingering issues that need to be tracked down on relatively unimportant files). I've created a script to create a tape to load into my 2.11BSD pl 195 to build, in a chroot, a 2.11BSD pl 0 system.

There's a script that I've build that builds everything at the pl0 level (twice). There's noise in the release notes of at least some of these releases that there reproducibility issues. It's currently past the initial bootstrap phase. I can build all the libraries, but automation is needed.

Following Along

There's two ways to follow along. One is to follow me on Twitter. My handle is @bsdimp. Or you can look at my github project. I've written up the status there (though it's a couple of weeks out of date) and you can find the start of a paper (though it's even more out of date, but has more background). I update at least once a week, but sometimes more as I have time.

Old-school Disk Partitioning

Old-School Disk Partitions

Unix started out life in 1970. Many of the things it did, it had to invent on the fly. Disk handling was one of those items. Something we take utterly for granted today was, once upon a time, an area of active innovation. This blog will explore the early days of Unix: when we had static partitioning compiled into the driver.

Typical drive of the era: large, and difficult to manage (an RL01 or RL02 by the looks of it good for maybe 5MB or 10MB of storage).

Evolution of Partitions

Unix from the earliest days had at least two types of data that was stored on disks. The first type was a file system, which offered a hierarchical name space. The second type was swap. Unix was a swapping system from the earliest days, and to multiplex jobs into and out of memory. The kernel would keep track of what blocks were assigned to which process.

Since the disk played two different roles, with different allocation policies and persistent storage. To keep things simple, in the first edition, the disk was partitioned in a static way. The file system used its part of the disk, and swap used the rest.

DEC never did produce a partitioning standard (though later it used Unix's on its Unix-related products). This mean that the Unix guys had nothing to draw from, and so failed to produce a standard before Unix left the research group.

1st through 3rd Edition

There was almost nothing resembling partitioning in the 1st-3rd edition kernels. Unix booted off the 'drum' device using the top 64 blocks to store itself. The rest of that disk was reserved for swapping processes into/out of core memory. The sources refer to this as 'the drum' and there were 1024 256-byte blocks in the drum.

The disks were presented as a device node, which was the entire disk. This is from the 3rd edition manual page:

rk? refers to an entire RK03 disk as a single sequentially-addressed file. Its 256-word blocks are numbered 0 to 4871. Like the RF disk and the tape files, its addressing is block-oriented.

which describes a single file that has its entire block store available. The 3rd edition has the enigmatic /crp filesystem documented for one of the drives (I wonder what it's an abbreviation for):

/dev/rk0	/ filesystem
/dev/rk1	/usr
/dev/rk2	/sys
/dev/rk3	/crp

which also shows that you had one filesystem per disk. At 1.2MB, these drives are little bigger than a floppy disk, so having just one filesystem per drive was not much of a limitation. Also note that the sector size was 256 words, or 512 bytes.

Since we have limited sources for this time period, it's hard to say for sure. The bulk of the surviving data, though, says at most the last 64 blocks are reserved for Unix...

4th Edition

With the 4th Edition, we start to see multiple files that refer to different sections of the disk. This was done because the new rp03 drives supported 81200 blocks, which exceeded the limits of the time of 65536 blocks per device. It also allows the drive to be broken up into more manageable chunks. In this release, there were 8 different files in the /dev directory named rp0 to rp7. For a second drive connected, rp8..rp15 would reference that drive. The manual has this table in it, which mirrors the surviving code:

disk	start	length
0	0	40600
1	40600	40600
2	0	3200
3	3200	39000
4	42200	39000
5-7	Unassigned

which allows basically two configurations for the drive: split in half, or split into a small root partition and two others. But where's the partition for swap?

There isn't one... Swap space in the 4th Edition was configured in param.h #defines. You defined the device, start and length on that device to use. The system would then configure that during early boot. The down side, though, of #defines was one couldn't easily do a binary patch. Swap was put in between the used parts of other of the disk. This was tricky to get right, since you had to map out the system.

You'd think that all the drives were like this, but that's not the case. The rf(4) driver specified the size of the drive itself with different minors. The rk(4) driver allowed one to experiment with different interleaf factors with different minor numbers. It really was up to the drive itself to decide how to interpret the minor number. This isn't unusual.... the various magtape drivers used minor numbers to specify density and whether or not to rewind on close.

5th and 6th Editions

The 5th edition continued the evolution. It changed the table above and there's a number of overlapping regions that need special care to be used. This also meant that you'd have to re-install the system, or hack the tables in the driver when you upgraded, if you used xp3 or xp4. There was only about 7 months between the 4th and 5th editions, so I suspect this problem wasn't too common. When 4th edition came out, there were 20 sites running it, by the 5th edition it was up to 50, so the numbers weren't huge. Since the kernel was patchable with adb or front panel switches, this issue likely could be mitigated enough to rebuild. Folks running 4th edition were trailblazers, by definition, so could be expected to cope with any upgrades they were doing to the 5th edition. By the time the 6th edition came out, it was in the hundreds or thousands, which may explain why it remained constant between 5th and 6th edition releases. The rp(4)'s predefined table became this:

disk	start	length
0	0	40600
1	40600	40600
2	0	9200
3	72000	9200
4	0	65535
5	15600	65535
6-7	Unassigned

Swapping was still handled by using the space between the partitions or after the end of the filesystem. The 4th edition way of configuring swap was improved by moving the defines to variables that can be more easily patched with adb(1). So through the 6th edition, swap still didn't need its own partition.

What is interesting is that 5th edition introduced raw character devices as well as block devices used for filesystem.

The 6th edition uses the same tables as 5th edition. It also introduces other disk interfaces with similar hard-coded partitions using a similar scheme.

7th Edition

The 7th Edition changes the layout again. It wasn't to be mean, but was because DEC introduced newer, larger drives. 7th Edition also allows larger partitions, as disk addresses are now 3 bytes instead of two. Some drivers now support multiple drives, so the partitions are fixed, but setup so that different layouts of the devices are supported, as well as different models.  For example, the hp(4) driver supports both the rp04/rp05 and the rp06 drives (which are twice as large as the rp04/rp05 drives) so has a number of different layouts that align to this size difference so they can be used on as many different drives as possible. The hs(4) driver was also added as well, but it was more of a swap / drum device.

The problem in the PDP-11 world is that new controllers have been appearing, some from third parties, and the number of disks you can connect to the PDP-11 has started to proliferate. This problem was only going to get worse.

2BSD through 2.11BSD

Skipping ahead a bit, the 7th Edition begat 2BSD which begat a series of releases for the PDP-11 (starting amusingly enough with the 'last' PDP-11 release: 2.8BSD!). By the time we get to 2.11BSD, we have a system that has about 50 different types of disks and partitioning schemes that are increasingly difficult to manage. Each one is a special little snowflake depending on what kind of drive is attached. The table in xp the driver runs 75 lines, which the authors know is bad because they prefix this table with:

/* THIS SHOULD BE READ OFF THE PACK, PER DRIVE */

 We'll return to 2.11BSD in a part 2.

4BSD

4BSD had a similar problem. They made it more palatable by creating a program, diskpart, which displayed the default disk partitions for a specific drive type, or allowed one to create partitions tables to cut and paste into a driver and/or config files. This helped, but was still compiled into the kernel rather than on the disk pack. 4.1BSD introduced, in 1981, a new wrinkle:

 * The bad sector information and replacement sectors
 * are conventionally only accessible through the
 * 'h' file system partition of the disk.  If that
 * partition is used for a file system, the user is
 * responsible for making sure that it does not overlap
 * the bad sector information or any replacement sectors.

Progress, but it was slow. Through at least 4.3BSD this was the case: every driver had their own table that was hand-tweaked for the drives that driver supported. This was OK, as far as it went, but as the industry shifted away from controller + disk combos where there were only a few choices to standard interfaces between the drive and host and most of the functionality in the drive, this became an untenable situation.

We'll explore how it got worse, before it got better in my second part. We'll see how different vendors innovated in this area ahead of a solution that appeared in 4.3BSD Tahoe and in AT&T System Vr3.0 around the same time.

[[ edited to correct typos ]]

Whither chroot?

Chroot Origins

This blog post will examine original artifacts to clear up some confusion about where chroot(2) and chroot(8) came from. The answer turns out to be simple, and the confusion was understandable. This shows the benefits of groups like TUHS in preserving Unix history, and how the kindness of Caldera and Lucent in releasing the historic Unix systems has helped in our understanding of the evolution of Unix. 

EDIT: After initially published, this was revised with more links to historic artifacts (inline and in the Appendix) and a screen shot of wikipedia. The Wikipedia chroot entry has since been updated.

tl;dr: chroot(2) came from 7th Edition Unix

chroot is system call 61 in 7th Edition Unix from Bell Labs. There is no chroot system call in 6th Edition or earlier. All derivatives of 7th edition have chroot(2) for at least 2 decades after the 7th Edition release in 1979.

What confusion?

Wikipedia has this in their entry for chroot:

which suggests that Bill Joy had something to do with its creation in the BSD world. Turns out it's confused because earlier literature on the topic is also confused.

What Sparked the Confusion?

Poul-Henning Kamp created the jail system for FreeBSD. This system takes a chroot environment to the next level in terms of security. As a security device, chroot was terrible because it's fairly easy to jailbreak out of a chroot if you are root. The short version is to open '/' to get a reference to it. Then chroot to some directory further down the tree. Then fchdir to the fd you saved from '/'. Now chdir(".."); a bunch of times. This will walk you back to the real root. Now chroot(".") and you are out. There's lots of variations on this theme, and dozens of papers in the literature and an almost infinite number of ways to leak references to FDs outside the jail...

One of the wonderful thing he did was to create an extensive set of docs and write a paper about the jail(2) facilities. In this paper Mr Kamp wrote:

[CHROOT]

Dr. Marshall Kirk Mckusick, private communication: ``According to the SCCS logs, the chroot call was added by Bill Joy on March 18, 1982 approximately 1.5 years before 4.2BSD was released. That was well before we had ftp servers of any sort (ftp did not show up in the source tree until January 1983). My best guess as to its purpose was to allow Bill to chroot into the /4.2BSD build directory and build a system using only the files, include files, etc contained in that tree. That was the only use of chroot that I remember from the early days.''

This paper was presented at the 2nd International System Administration and Networking Conference "SANE 2000" May 22-25, 2000 in Maastricht, The Netherlands and is published in the proceedings.

In 2000, the BSD SCCS tree was not publicly available. Dr McKusick had access to it as his role with the Computer Science Research Group (CSRG) that produce the 4BSD releases. This predated various litigation that suggested 32V had no copyright, and the Ancient Unix License that SCO granted for 32V, so it was necessarily private per agreements between AT&T and The University of California at Berkeley.

What Actually Happened in 1982?

What happened was a shuffling of the deck chairs. the commit log, made as root, from March 18, 1992 says:

rearrange for kirk

SCCS-vsn: 4.21

and introduces chroot to ufs_syscalls.c. If you read the diffs, it also introduced 'open', 'creat' and several others to this file. These system calls are known to be in the PDP-7 Unix implementation, so it's unlikely that they were really introduced in this commit. One problem that makes this harder to track is that SCCS didn't track renames, and ufs_syscalls.c was renamed to vfs_syscalls.c in 4.4BSD.  It's quite clearly in ufs_syscalls.c in 4.1cBSD:

/*
 * Change notion of root (``/'') directory.
 */
chroot()
{

        if (suser())
                chdirec(&u.u_rdir);
}

which is the identical code that was added by Bill Joy to ufs_syscalls.c. This was moved between 4.1BSD and 4.1c from sys4.c as part of the UFS work, and is different only by the BSD-stylistic change to add a blank line before the rest of the code if there's no local variables:

chroot()
{
        if (suser())
                chdirec(&u.u_rdir);
}

which, apart from the comment, is identical. Without beating a dead horse (too late?), this code is the same all the way back to 4BSD, 3BSD, 32V, 2.8BSD and finally to V7:

chroot()
{
        if (suser())
                chdirec(&u.u_rdir);
}

Since the code is identical from V7 all the way through 4.2BSD when it was, according to this footnote in the jail appeared, added. This is direct evidence that the footnote was in error.

So what was the rearrangement for Kirk? It was to move things around in the kernel to make the system calls more generic. It was code motion, nothing more, that Dr. McKusick was reporting in the private email to Mr Kamp. Now that the SCCS tree is public, via a translation to svn by John Baldwin, we can see the above.

chroot(2) Conclusions

Given that the code was moved around alot, it's an understandable mistake that Dr. McKusick made, which explains how the error could have happened. Given that the code is identical to v7 code, and it was somewhere in all the extant versions between the two (2BSD, 32V, 3BSD, 4.0BSD, 4.1BSD, 4.1cBSD and 4.2BSD), modulo a trivial whitespace change, we can conclude that Bill Joy did not introduce chroot into 4.2BSD, but instead it was moved around a lot from the original V7 code.

The FreeBSD chroot(2) manual has been updated to correct this mistake.

But what about chroot(8)?

But what about chroot(8)? There's some confusion about this as well. Until recently, chroot(8) said in FreeBSD:

HISTORY

     The chroot utility first appeared in 4.4BSD.

However, that too is in error (or was at least not precise enough). The error comes from the 4.4BSD release itself, which has identical text. In a sense this is not wrong. 4.4BSD was the first full release that chroot(8) appeared in in the Berkeley world. It's first appearance, though, in any BSD tape was in the interim 4.3BSD-Reno release.

But what about the AT&T world? There, more system calls are wrapped in programs to make it easier to use in shell scripts. It turns out that System III had a usr/src/cmd/chroot.c, which I won't quote here, that's a different chroot than appeared in BSD (the code looks completely different, apart from the elements that have to be the same...). So, the history has been corrected to read:

HISTORY

     The chroot utility first appeared in AT&T System III UNIX and

     4.3BSD-Reno.

to represent the first time in each of the two branches of Unix after the 7th Edition that it appeared.

And that concludes today's software archeology deep dive on chroot...

Appendix

Here's the evolution of the chroot(2) implementation, as see from TUHS. You'll need to search for 'chroot()' in each of these source files since the current TUHS web site doesn't allow line number links.

AT&T Unix: V7, 32V, System III, System V

BSD: 2.9BSD, 3BSD, 4.0BSD, 4.1BSD, 4.1cBSD, 4.2BSD, 4.3BSD, 4.4BSD.

I'd also like to plug the Historic Unix Repo, which also helps navigate and allows line numbers. Here's a link to the 4.1c version, for example. I recalled this after I'd found all the TUHS references, or I'd done all of them like that.

Adding a second disk with SIMH and 2.11BSD

Adding a Second Disk to a 2.11BSD system under SIMH

I recently followed some instructions to get 2.11BSD running under SIMH. That topic is covered elsewhere adequately. I may write something up in the future.

Before I started, my simh.init file looked like this (some items from install omitted)

SET CPU 11/93, 4M
SET CPU IDLE
SET RP  ENABLE
SET RP0 ENABLE, RP06, WRITEENABLED
ATTACH RP0 ./2.11BSD
SET XQ ENABLED
SET XQ TYPE=DEQNA
SET XQ MAC=08-00-2b-11-07-82
ATTACH XQ tap:tap0
; At the SimH promp type: unix
BOOT RP0

As part of my 2.11BSD patch level 0 restoration project (more on that later), I needed to add another disk I could install chroot images to test building. I'm running 2.11BSD pl 457 at the moment (I've not walked forward form the last snapshot tape). Fortunately, this version has disklabels, so I'm able to do this the easy way (though the old hard-coded stuff isn't too hard either).

First, I needed to add the raw disk in simh. I opted to have a second RP06 for simplicity. There's adequate space. My 'root image' for 2.11BSD I want to test is about 100MB, and the RP06 is 165MB. That should be adequate. I just needed to duplicate the RP0 lines:

SET RP1 ENABLE, RP06, WRITEENABLED
ATTACH RP1 ./extra-data

and restart simh (be sure to halt any running system before stopping simh). The important part here is to configure RP1 as writeable and an RP06. Otherwise, it will default to the smaller RP04. For me, that's too small.

Next, I had to check to see if there were /dev nodes for this device. The xp driver handles RP06 (and many other) disks.

3% root-> ls /dev/xp1*
/dev/xp1a  /dev/xp1c  /dev/xp1e  /dev/xp1g
/dev/xp1b  /dev/xp1d  /dev/xp1f  /dev/xp1h

so I'm in luck. The devices are there. Otherwise I'd have to run /dev/MAKEDEV in /dev to add them (or worse, do it by hand).

Next, I needed to label the disk. It's fortunate I'm running a new version because this was easy and I didn't have to rely on the hard-coded partitioning in the driver. However, even if I did, I'm using the whole disk so it wouldn't change my life much...

5% root-> disklabel -r -w xp1 rp06

which puts the standard rp06 label (from /etc/disktab) onto the drive. Chances are good that this will work on older versions. This gives the following label:

6% root-> disklabel -r xp1
# /dev/rxp1a:
type: unknown
disk: rp06
label:
flags: removeable badsect
bytes/sector: 512
sectors/track: 22
tracks/cylinder: 19
sectors/cylinder: 418
cylinders: 815
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # milliseconds
track-to-track seek: 0  # milliseconds
drivedata: 0

8 partitions:
#        size   offset    fstype   [fsize bsize]
  a:     9614        0   2.11BSD     1024  1024         # (Cyl.    0 - 22)
  b:     8778     9614      swap                        # (Cyl.   23 - 43)
  c:   153406    18392   2.11BSD     1024  1024         # (Cyl.   44 - 410)
  d:   168724   171798   2.11BSD     1024  1024         # (Cyl.  411 - 814*)
  e:   322130    18392   2.11BSD     1024  1024         # (Cyl.   44 - 814*)
  g:   171798        0   2.11BSD     1024  1024         # (Cyl.    0 - 410)
  h:   340522        0   2.11BSD     1024  1024         # (Cyl.    0 - 814*)

Note one difference from modern FreeBSD: rxp1a. 2.11BSD still has the character/block split. Also the 'standard' layout looks a bit odd to modern eyes. But there's 4 sets of partitions here: a,b,c,d for a system disk with / on a, swap on b and /usr on c. a,b,e (same but with a larger /usr on e). g and d to split the disk in half for data storage. And h for the whole disk. This mirrors the partitions from when things were hard coded in the device driver (yikes! glad we don't have that legacy anymore). In those days, you had to be as flexible as you could and leave it to the sysadmin to make wise choices with the limited flexibility they hard. These days, I'd label a scratch disk with just one partition (and call it 'a'). Since I was being lazy, I thought I'd leave this label in place. It's a quaint curiosity, but also instructive of history.

So, next, I have to put a filesystem on it. That's done with newfs:

8% root--> newfs /dev/xp1h
newfs: /dev/xp1h: not a character device
9% root--> newfs /dev/rxp1h
newfs: /sbin/mkfs -m 2 -n 209 -i 4096 -s 170261 /dev/rxp1h
isize = 42560
m/n = 2 209

which gives me a new filesystem. This is quite a bit less chatty that I'm used to on FreeBSD. Also, even after noticing, I forgot you have to newfs and fsck the raw device, not the block device.

Now time to mount it and add it to fstab. Old-school write ups say to fsck /dev/rxp1h here, but given simh doesn't simulate the unreliability often found in the hardware of the time, I've skipped that part.

10% root--> mkdir /scratch
11% root--> mount /dev/xp1h /scratch
12% root--> vi /etc/fstab
"/etc/fstab" 3 lines, 79 characters
/dev/xp0a       /       ufs     rw              1       1
/dev/xp0b       none    swap    sw              0       0
/dev/xp0c       /usr    ufs     rw              1       2
/dev/xp1h       /scratch ufs    rw              1       1

I've scrunched the vi session into the above: I just added the last line. And now I have a /scratch filesystem that will survive reboot.

And now I'm ready to create a tape with my putative 2.11BSD pl 0 system (really at the moment a 2.11BSD pl 195 system with pl0 sources). But that's for another day.

FreeBSD's METALOG: unprivileged installs

What is METALOG?

When you 'make installworld -DNO_ROOT DESTDIR=blah', the system will create a $DESTDIR/METALOG file. This file contains all the permission and modes for the files. Normally, installworld requires root permission. -DNO_ROOT instructs the build system to install them as the user and to note what permissions, etc in a METALOG.

How to use METALOG

Creating a UFS partition with no privs

If you have your own tooling around image creation, you can use the METALOG to supply the permissions and other filesystem metadata to that process. makefs can be used by a non-privileged user to a UFS partition image. Coupled with mkimg, you can create an entire bootable system image without needing root. Look at the -F flag to makefs(8) for how to use this functionality.

Package Base Use

METALOG is also used by the pkgbase initiative to slice up the system. Part of the metadata that's included is what package each of the installed files belongs to. This is all transparent when you do a 'make packages' to generate these packags.

Taring up  an installworld

If you are looking for a quick and dirty way to udpate a VM, you can often just create a tarball from the METALOG. Tar was enhanced a number of years ago to understand mtree files. The METALOG is one giant MTREE file. To create a tarball that's a copy of the image with all the right permissions:

cd $DESTDIR
tar cfJ base.txz @METALOG

This will create a xz compressed base.txz similar to what the release images create. This one tarball has everything (unlike the base.txz from the release build process), and is about 800MB.