Some Hints For Splitting Commits

Sometimes in a code review, the commentators suggest that commits be broken up into smaller pieces. Here are a few of the collected tricks I have learned over the years. They are presented as 'editing the current tip of the tree' for simplicity. However, you can apply them to rebasing as well using the 'edit' action. The last section offers a strategy when you need to recombine the commits.

Splitting up

Sometimes you have a commit that you need to split up. This section will show you how to do that. Briefly, you will reset the HEAD of the tree to remove the commit from the current branch (the original hash remains in the repo), and then add and commit it piecemeal.

Before we get started on doing the change, there’s two steps to do:

• Run git status to ensure the tree is clean.
  If the tree is unclean, then you can lose changes in the following steps (or have them accidentally merged in).

• Run git show to show the commit you are working on.
  This is especially important when using these steps when doing a git rebase -i with and edit step.
  It also helps to know what the whole change looks like.

Now, to 'undo' the change from the repository, make sure the changes are what we think, then redo the changes.

• Run git reset HEAD^ to 'undo' the change.
  This will set the pseudo-tag ORIG_HEAD to the value of HEAD before this command.

• Run git diff and ensure that the diff matches the diff part of the git show you did earlier.

• Run git add -ip to interactively select the subset of the change you want.
  There are instructions for how to edit the context diff at the end of the diff, so I won’t go over them here.

• Run git commit -c ORIG_HEAD to commit the change.
  This commits the change and copies the commit message fro the pseudo-tag ORIG_HEAD.
  More often than not, when you are splitting changes, you will want a subset of the original message, or to edit it somehow.

• Run git diff to see what’s left to commit.
  Repeat the last two steps until there’s no more changes left.
  git commit does not move ORIG_HEAD so you will start with the right commit message each time.

If you are really operating on the last change in your patch series, you are now done: the patch is split up. If you are doing this inside a git rebase -i with an edit action for the step, do not forget to now run git rebase --continue to complete the process. I usually just edit one at a time to keep things simple (I often forget to do the git rebase --continue if I have too many things to edit).

Rearranging the commits.

Sometimes, you have 2 commits that should be 2 different commits, or some similar rearrangement. Each of the new commits have parts of the original commits. What I like to do here is take the original two commits and break them down into the basic parts using the previous section. So I break the first commit down into the commit into 2 new commits. I repeat the process for the second commit. There are now 4 commits in my branch. I usually keep the original commit messages intact and do not edit them at this stage. I then use git rebase -i to recombine them using the 'squash' action. This gives me the new commits with copies of the original two commits' commit messages so I can edit each one down appropriately. I find that doing this in multiple steps makes it easier to keep track of everything. This allows easy backing out half way through if you realize you have done something in error. Doing only one or a few things at a time makes it easier to do that. git reflog can also help if you made a mistake several steps ago.

DEC Rainbow Floppies Found on EBAY

 DEC Rainbow Floppies on EBAY

A group called "Australia SDC" recently liquidated their DEC Rainbow floppy collection on EBAY. I won the auction and will be placing them up once I have them imaged. There were a lot of application floppies that I'll do a separate blog post about (about half of the lot are programs like tk-solver, 20/20, MT Pascal, CBASIC, etc)

However, interestingly they had the following floppies that contain drivers for 3rd party software:

• DM220 Disk Utilities -- Drivers for Duncan Mac Donald products (Version 1.0 4/14/1986 MS/DOS)
• DEC RAINBOW MS-DOS 2.05 for ACT WINCHESTER (believed to be a boot disk with the ACT Winchester driver included, ala the Univation disk controllers that did a similar trick)
• DEC RAINBOW 100 CP/M-86 for the ACT Winchester v28.09.84
• C.H.S. Corp XDRV.SYS Version 1.0 Rainbow MS-DOS format (this is for CHS's dual hard disk controller)
• Rainbow Diag Disk (8087 co-processor and extended memory BL-DE29A-BV
• O.P. SYSTEM For Rainbow Winchester Utility/Diag Disk BL-W968B-M3
• Winchester Utility/Diag Disk DOS 2.11 (BL-w968B-M3)
• ZV0003.M3 V2.0 util/diag disk (hand written: R-Bow Hard disk Util) BL-W968B-M3
• Winchester Utility/Diag disk BL-W968B-M3

The last 4 appear to be the same disk, but they have radically different labels, so it's unclear what else might be on the disks.

There's about 20 unlabeled RX-50s in the cache as well. One of them says 'backup 1986' on it, but no clue if that's a complete backup, or just someone's hack to xfer something. I'll do a blog on that as well.

The Duncan Mac Donald products diskette is interesting, since I've since ads for his products for the Rainbow, but no actual drivers. The ACT Winchester is interesting. I'll have to look at that in detail as well. I have another copy of the CHS diskette, and the driver is included on SS MS-DOS 3.10b disks that I have, but you never know what else might be there.

all in all, an interesting find.

DEC Rainbow Power Supply Specs

 DEC Rainbow 100B Power Supply Specs

For the next time I'm looking for this... The DEC Rainbow power supply, as documented in the TM100 technical reference manual, is:

Output: +5.1V +/- 6% 2.5-11.5A, +12.1V +/- 6% 0.6-6.7A, -12V +/- 7% 0.0-0.15A

Input: 115V 6A (218W)

Pinout:

Pin Function

1 ACOK H

2 DCBIAS (unused)

3 Key

4 -12V Input

5,6 +12V Input

7,8,9 +5V Input

10,11,12,13 Ground (DC Return)

Most ATX power supplies supply more 5V and less 12V than is needed for the Rainbow to power the system. However, the 12V supplies are in large part for the older disk technology in the Rainbow and if you are using emulators you can get by with a little less, though how much less is TBD.

Spelling Fixes -- Some Advice

 Some Thoughts on Spelling Fixes

I've been looking at a number of FreeBSD pull requests lately. Many of them are spelling fixes. I'd like to offer some suggestions for people submitting them. The same advice applies to typo fixes and grammar corrections.

1. Keep number of changes small.
2. Use separate commits per directory
3. Use descriptive commit messages
4. Set your email correctly
5. Don't correct code

Keep number of changes small

When submitting like this, limit the number of changes to 10-20. More than about 15 changes becomes hard to review. Every single change has to be verified for correctness, and having too many will make your pull request more likely to be overlooked.

Use separate commits per directory

When submitting a number of changes, do one change per subdirectory. When subdirectories are nested, it's OK. For example, if you have changes to bin/ls and bin/rm, do two commits. But if you have changes to both usr.sbin/wpa/wpa_priv and usr.sbin/wpa/wpa_cli, then it's OK to do those as one commit.

Use descriptive commit messages

The commit message "fix spelling" is too generic to be useful. If you are fixing just one word, a better commit message would be "Spell interrupt correctly". It also allows the reviewer to make sure that you are changing the right thing. And it also gives enough detail that people skimming the logs don't need to look at the diffs to know what changed. If you are fixing multiple spelling errors, then a generic message is more appropriate.

Set your email correctly

When you push your branch to github, make sure that you've set the email you want in the commit message. It saves a lot of time. If you are using github's editor, make sure that your profile has this information set correctly.

Don't correct code

Don't make spelling corrections in code variables, #defines, etc. These will likely be ignored. The risk from a comment or an error message being corrected is tiny, while code changes could be attempts to subtly change the system or introduce security impacting issues through a supply chain attack. It's better to work with someone in the community to get these corrected than to correct them via a pull request.

A new path: vm86-based venix emulator

 Venix Emulator Update

It's been a while since I've had time to work on the Venix emulator. When I set it aside over a year ago, I'd taken it as far as I could with the 8088 emulator I'd found online. It had no FP emulation and there were

a number of things misbehaving that I couldn't quite get right. And exec was proving hard to implement. Despite being written in C++, the original emulator resisted my efforts to make multiple instantiations.

So I set it aside last May, thinking I might get back to it when the qemu bsd-user changes FreeBSD has done have been upstreamed.

The first of August I took some time off from work and got the bsd-user changes in shape to upstream. Well, the first 10% of the changes that were the hardest since it was replacing what was there with something that minimally worked. This helped me learn qemu's x86 CPU much better and it got me thinking that qemu's user-mode stuff might be the way to go.

About this time I also found a vm86 test program in the FreeBSD tree. So I got to wondering, could I do a vm86 implementation of Venix?

So, I stole the bulk of my old 86sim-based Venix implementation, installed a i386 VM using bhyve on my FreeBSD/amd64 box and write a quick little test program. The test program worked, so in a fit of "why not give this a try" I ported the pcvenix.cc from 86sim to being driven from SIGSEGV in vm86 mode. Hello world quickly worked.

Enter VM86VENIX

So, I reworked fork and exec and the a.out loader a bit. I was able to get the C compiler going in this new setup. The 'cc' command is just a fancy script that strings together the pre-processor, compiler, optimizer, assembler and linker. Except on Venix it wasn't a shell script I could hack to run natively on FreeBSD. It was this weird binary that did all the forking, execing, redirecting, etc inline. More on that in a minute.

So vm86 mode is a special mode in 32-bit CPUs that lets you execute old 16-bit code in the context of a normal process. It's super easy to setup, but often of limited use.

Thankfully, the i386 ELF designers thought ahead. The starting address for binaries in ELF is this weird 0x00401430, which is just above 4MB. This means that one can map anything into low memory and it will work. FreeBSD has a security stop on mapping anything at location 0, however, but the rest of the first 4MB is available. The old 8086 could only see the first 1MiB of that, but since Venix binaries are at worst 'small-mode' the largest address space for a process was 128kiB. Plenty of room to find a place to map it.

So, I wrote a loader that would load the old Venix a.out binaries into this space. Or rather I hacked the loader I already had to do the mapping. I was able to reuse all the loader code from the 86sim-based emulator I had before.

I then shamelessly stole the setup code from the FreeBSD vm86 testing binary, which was little more than establishing signal handlers and zeroing the context and setting up a stack and IP as well as the segment registers. With that in hand, I was able to use sigreturn() to set the processor flags such that it would jump to where I wanted to go in the Venix binary. I'd been afraid of vm86 mode after reading through doscmd years ago, but there was no need for the fear: all the cruft in doscmd (I reread it after this) was the accumulation of cruft over the years for DOS, BIOS and other weirdness that evolved around the IBM PC, XT, AT and the plethora of clones which had nothing to do with vm86 mode, per se.

Every INT xx instruction would trap to the kernel. The kernel would note down the registers and send the process a SIGSEGV for these accesses. I was able to then look at CS:IP in the mcontext and decode the instruction that faulted. INT xx is encoded as the bytes 0xCD 0xXX for almost all values of X (INT3 has its own opcode 0xCC). In the signal handler, I could decode this opcode. I knew from past work that INT 0xf1 was the system call, so I hooked up the old venix system call handlers to this and I was back to where I was with 86sim. Further in fact because floating point worked.

Signal handlers have an implicit sigreturn with the context passed to the signal handler at the end. I needed to skip over the faulting instruction after performing the system call, and the process would then resume executing in 16-bit mode after the INT 0xf1. This was straight forward to implement.

I decided to implement fork as a real fork. It would copy the address space, all the open FDs, etc. This proved to be an easy way to cheat so I didn't have to create a context object and use threads to simulate processes.

Exec proved to be just a call to my loader that started all this off. The only thing I've not implemented is close on exec, but the rest was easy.

With these implemented, I could run the C compiler's cc command and generate trivial binaries. But if I needed to include anything, it would fail. I created a VENIX_ROOT env variable. For all opens, it would try VENIX_ROOT / name and then just the plain name if the name arg to open started with a '/'. This was enough for the preprocessor to include .h files and for the canonical hello world to build.

There was just one vexing problem: cc -o hello hello.c worked. However cc -O -o hello hello.c didn't.

Tracking down a silly bug

Well, there was another annoying thing: /bin/sh didn't work. I traced that to the fact I've not implemented passing an environment to the processes, and /bin/sh was choking on that. OK. Fine. I'll implement that later. /bin/csh worked, however, so I was happy. My happiness was short lived, alas, because I'd run a  command and I'd get weird output:

% ls
ls: Sig 44
%

 That's weird. So I added tracing. I tried the cc command, which in this version is a simple program that orchestrates all the different parts of the compiler using fork, exec, dup and the strategic close/open pair to setup stdin etc. All the tracing looked good as well, we'd see something like:

123: fork() 124
123: wait()
124: ... lots of stuff
124: exit 0
123: wait pid 124 status 0

 and then 123 would proceed to delete all the temp files and exit. It was like it was getting an error, despite its children exiting without an error. Every time it was like this, but only when I ran the optimizer. When I'd re-run the ls test, I'd get different Sig values as well.

Available 8086 compilers in 1985

So, I'd assumed that the compiler was derived from the V7 compiler. However, a number of hints in names suggested it wasn't. And the Venix manual had way more exceptions for 8086 than for pdp-11 when it described the options and operation for Intel, so I assumed it wasn't V7 derived. So I started hunting around for C compilers.

MIT produced one at the time for the PC. It could run on the VAX and generated a.out binaries that a conversion program would convert to .COM or .EXE files. I thought this might be where the Venix compiler came with. But after playing around with it for a few hours it was clear it wasn't. First, it had a shell script cc, not a program. Second, it had a number of VAX specific instructions sprinkled inline, and that wasn't going to run on the Rainbow :).

I took a look at the portable C compiler. This I think was the real genesis of what was shipped with Venix, but old versions that support 8086 are hard to come by, even in the successor portable C compiler project that's been going for 20-odd years now. I got the cc program from that compiling with Venix. It was a bit easier to fuss around with than the V7 one (but the V7 one would be close enough to hit the bug I found out later). Looking at the old System III sources that one can find on the internet, there's a copy of the portable C compiler there, rather than the C compiler from DMR as you'll find in the 7th edition. I used the cc program from there to try to build things. I hit the jackpot: it failed faster!

So I instrumented the pcc program and discovered that the status printed after wait() in the program didn't match the status that I'd returned from the kernel. Progress!

The wait(2) system call...

I'd implemented the wait(2) system call as part of getting fork/exec working. I did it from the VENIX manual that's available online. I looked at the first part of the manual:

SYNOPSIS
        wait(statusp)
        int *statusp;

which shows wait taking a pointer. So I assumed this was what the kernel received in the first arg that's passed into the kernel (DX). I assumed that DS:DX pointed to an integer where I'd return the status. Most of the time, DX was something that looked like a pointer on the stack, so I just did a copyout. The problem is that's not right.

So, I took another look at the manual. At the end of the man page I saw:

8086      BX=7
              int 0xf1
              AX = pid of process
              DX = status

and then it hit me.  DX isn't a pointer to anything. After int 0xf1 AX is the pid of the process (the normal return value) and DX is the status. Disassembling wait.o confirmed this:if statusp is not 0, dx is copied back to *statusp. Doh! The classic pointer vs value mistake. Fixing my implementation to take out the copyout and replace it with setting DX in the processor context made pcc work. And cc worked. And the silly test programs I wrote in the middle to debug things worked. Woo Hoo!

Once I fixed this, all weird combinations of compilations suddenly worked for me. I could optimize, strip, etc and there were no oddities.

Conclusion

It helps to read the manual carefully!

I need to try to build the system. There's shell scripts to do that that don't depend on environment variables working if run natively, so I'll see if they work and see how much of the system I can generate via this route. Stay tuned.

My TODO list still contains getting env working (I don't think it is hard, but I think I need to filter things because my default env is larger than the stack on these old x86 machines). I also need to look at rebasing my emulator as a *-user qemu emulator (even if they don't take it upstream). Maybe even add PC/IX and Xenix/86 support as well so that other researchers can play around with this.

On Updating QEMU's bsd-user fork

 QEMU bsd-user

bsd-user is a 'user mode' emulation tool. It emulates FreeBSD's system calls on FreeBSD well, and $OTHER-BSD system calls elsewhere to varying degrees of success. It's primary mission has been to build FreeBSD packages using user-mode emulation to speed the process over using system mode. It speeds things up because the compilers and other huge CPU hogs can be built natively.

Of late, it has languished, A few years ago, I started to rebase it to the then-tip of qemu in the hopes of upstreaming. At the time, we'd forked off of qemu 1.0 or so. During this time the then-current qemu was 4.0. I got things rebased to around 3.1 before running out of steam. Rebasing patch trains of 1000 commits is hard, and trying to selectively squash commits wasn't much better.  So that's where things stalled. All bug fixes to qemu bsd-user had gone on in our own private branch.

Recently, I'd been asked about it again, so I started to dust things off. I got my name listed as the maintainer so I could push patches upstream a little more easily, and then started contributing by doing basic cleanup in the hopes of redoing 'logically' what had been done to split things up. Those efforts too have come to naught.

So, in one final act of desperation, I copied the 3.1-rebased bsd-user directory directly into qemu 6.0 and got it building. There were lots of little changes I needed, but nothing super huge. I've not done extensive testing, but the basics seem to work.

Trouble was, that diff was 35k lines. Too big to upstream in one go. So, I set out to see what could be done.

First, I labeled the 'yeet it up  to current' branch as 'blitz'. It's a fast hack to get something we can move forward on. In the future, releases and such will be cut from there until I can get it into the upstream tree. Blitz is the German word for 'fast' and has connotations of doing something quick and dirty well enough to move on.

Next, I created another branch from 6.0 called 'kaizen'. Kaizen is the Japanese business practice of continuous improvement. Find the most painful or most expensive part of your business, fix that and iterate. This branch I'll be putting 'diff reduction' patches for upstream, as well as start to move things over from blitz, starting with the loader. I've disconnected everything except x86 from this branch. In upstream qemu, bsd-user core dumps right away, so I'm not turning off anything that's working.

So the plan is that I'll focus on keeping x86 buildable, and get it working as quickly as I can and then add all the system calls from the blitz branch. I'll add them one group at a time, and do the reorgs and new file creation as well. I'll get these reviewed and upstreamed. Once all the system calls are in place, I'll start adding additional architectures as well, getting those patches reviewed too. Finally, I'll get the NetBSD and OpenBSD hosting stuff updated, as well as take a stab at updating their system call tables and seeing how well it works. The work that Stacey Son and others did tried to preserve all this, but it's been a long time since any of it was tested.

I have an agreement in principle with the qemu upstream to do all this work. So approximately monthly, I'll be landing a new branch with the latest diff reductions. I'll rebase kaizen and blitz after each drop and before I upstream. For the moment, this work will go into my gitlab fork (since it has all the CI setup on it) and from time to time I'll publish back to the github qemu-bsd-user repo. Be advised: both the blitz and kaizen branches will rebase often, so you may need to do weird things to update. Though, if you are tracking them with changes, please be in touch so we can coordinate work.

With luck, by this time next year, the kaizen and blitz branches will be nothing but a distant memory and we'll be on to keeping things up to date in qemu head, maybe with doing some refactoring with linux-user where it makes sense.

 Customizing Emacs for Git Commit Messages

I do a lot of commits to the FreeBSD project and elsewhere. It would be nice if I could setup emacs in a custom way for each commit message that I'm editing.

Fortunately, GNU Emacs provides a nice way to do just that. While I likely could do some of these things with git commit hooks, I find this to be a little nicer.

First up, we need to do something when we pull up the commit message to edit. By convention, git uses the file COMMIT_EDITMSG, though the exact location of this file depends on where the git tree you have checked out is. Emacs has a hook that's run when emacs starts editing a file called 'find-file-hook'. So:

(add-hook 'find-file-hook 'imp-git-hook)

will do the job nicely. Next up, we need to define this function to do something useful when run (indeed, failure to define it will result in an error when visiting all files).

(defun imp-git-hook ()
  (when (string= (file-name-base buffer-file-name) "COMMIT_EDITMSG")
    (freebsd-git-setup))) 

buffer-file-name is a local variable that has the full path name to the buffer, if any. file-name-base is like basename(1) in that it returns the name of the file without the extension, rather than its whole path.

But what is 'freebsd-git-setup'? It's a little function I wrote to set the fill column to 72 (I usually have it at 80, but that produces commit messages that are just a bit wide when git adds leading spaces) and adds a sponsorship tag to my commits:

(defun freebsd-git-setup ()
 (save-excursion
  (setq fill-column 72)
  (if (re-search-forward "^Sponsored by:" nil t)
    t
   if (re-search-forward "^\n#" nil t)
    (replace-match "\nSponsored by:\t\tNetflix\n\n#")))))

But it only adds the sponsorship tag if one isn't there yet.

This is a proof-of-concept function. No doubt it will have to evolve over time. The project adds 'Differential Revision' tags as the last tag in a commit message because differential requires (required?) that. And it wouldn't be good to add this for non-FreeBSD commits, so I'll have to find a way to filter for that... But I thought this would be useful, if nothing else than to my future self as a roadmap for how to do things like this.