Backing up git repos on multiple machines to central repo w/o collision

Git Tree Management

Here's a quick column about git. It's not a complete how-to or tutorial, but more an interesting way to manage multiple trees.

The problem: I have a dozen trees on a half dozen machines. I'd like at least backup all the branches in these trees to github. Trouble is, I don't want branch names to step on each other. This can happen for a number of reasons, let's say I called something 'junk' by habit on N trees and don't want a push to screw that up...

Git's world view: To understand git, you have to understand that it is a graph of versioned trees with labels. Each node in the tree has the familiar hash, and some of the hashes have refs that the git branch command groks. It's all just a directed graph with labels under the covers.

Normally, you when clone a repo, all its tags magically change from foo to origin/foo (for some value of origin).

Enter refspecs

Turns out this has been thought of before. The answer is simple:
fred% git push origin foo:fred/foo
will push the foo branch to your origin and rewrite its name to fred/foo. Don't forget to push master too.

Now when you go to barney and fetch, you'll have a bunch of remote branches named origin/fred/foo, etc.


Since I'm doing this with a number of git svn trees, the cost is kinda high since git svn creates new, unique git hashes for all the upstream revisions to git svn rebase. It also means that you'll need to learn how to use the --onto arg of git-rebase, since if you want to move a branch from one repo like this to another.
barney% git checkout -b foo fred/foo
barney% git rebase -i fred/master foo --onto master
since you're effectively creating a new name space on your local machine for the new branch. The rebase will now properly take just those commits from foo, and then play them back onto master on the current machine and leave you with a 'foo' branch for the results.


Most things in the VENIX emulator are working


So I got tired of the terrible progress I was making chasing down issues. I thought if I could just create a simple program and get that working, I'd have much better luck.

So I wrote a simple K&R style C program:
int a=123;
int b;
extern char *etext, *edata, *end;
main() {
int c;
printf("CS: %x DS: %x ES: %x SS: %x\n", getcs(), getds(), getes(), getss());
printf("&data = %x &bss = %x &stack = %x\n", &a, &b, &c);
printf("etext = %x edata = %x end = %x\n", &etext, &edata, &end);
which printed the segment addresses and then locations of the segments.

and I ran it on my old Rainbow running Venix.

I made some interesting discoveries. First, that there are two kinds of stacks (low and high) in addition to there being two kinds of binary (OMAGIC and NMAGIC). So my loader was all wrong. Next, I discovered I needed to jump to a_entry, not 0, to make low stack binaries work (all the ones I'd been testing so far were low stack, but somehow mostly worked when the stack and text segments were swapped).

Armed with this knowledge, I built 4 binaries (no flags, -z, -i, -i -z) to test all 4 cases. The -z ones worked (yea!) while the non-z ones didn't. My loader was right in this case, but I was returning EFAULT for all the writes. Why? Because I had a check in there to make sure the address was between 0 and brk. High stack binaries also have a valid area from sp() to 0xffff. When I added that change, all 4 test programs worked.

Of course, getting them from the Rainbow to the server was a challenge. The key to remember here is that you needed to use 'set line /dev/com1.m' on the rainbow so that kermit would on login port. I also had to down-clock to 2400 baud to get it reliable.

So, I started testing a lot of programs that failed to work before. Sort(1) is now working. ls isn't, but comes closer (it tries really hard to interpret a modern FreeBSD dirent as a v7 one and that's not so good, but that's fixable). nm is still giving me problems, for reasons unknown. I have enough things working, though, that I can start to try out as, ld and friends. Maybe even cc (though I'd need to get both fork and signals working for that driver program). /bin/sh fails missing dup() (and likely a bunch of others).

So excellent progress in the last few days.


Even more VENIX emulator progress

So in looking at the traces for why cal wasn't working, I noticed something odd:

0212:100D: jmp 0x109e
0212:109C: rcrw $0xff,0xdceb(%bx,%si)
Invalid opcode c1
What? I'm not super-duper strong on Intel assembler, but I sure know that 109e is not 109c. So what's going on here. After adding some more debugging, I discovered this was opcode 0xe9, which is a jump relative with word (so take IP and add the next two bytes to it). So, the code looked OK:
                doJump(ip + fetchWord()); 
But looking more closely. It's oddly off by 2. Sow what's inside fetchWord()? Inside it effectively does ip++, twice. So, on the other compiler that was used for this code that I obtained from tkchia's reenigne repo had this flaw. it did fetchWord() + ip, rather than clang's ip + fetchWord(). So the fix was simple:
                t = fetchWord();
                doJump(ip + t);
which made the order of operations well defined. A quick audit of the code shows no other places where this is done.

I did that, and nothing else, and now cal(1) works. It produces correct calendars. As does od(1), uniq(1), pr(1) and others.

I've also implemented alarm(2), signal(2), lseek(2) and pause(2). With that, sleep(1) works (although it says 'Alarm clock' which suggests I need to actually establish a SIGALRM handler).

There's enough working I'm starting to need some kind of regression suite to make sure I don't regress and can publish the status of all the binaries... Maybe I could leverage an existing something...


VENIX/86 emulator taking shape...

Frequent readers will recall my obsession with Venix on the Rainbow.

For the past year or so in my off moments, I've been trying to put together a Venix binary emulator. This is part of a larger project to reconstruct the Venix sources from the ancient V7 sources plus clues left behind in various images found on the internet in time for the 50th anniversary of Unix next year.

Early in the summer I had the loader written. I could successfully load a VENIX image and start it executing. sync(1) was working, but it did little more than call the sync(2) and _exit(2) system calls. stdio programs were still basically not working, though odd things like ln(1) also worked.

My project is now one step closer to fruition. I have been able to get some of the basic programs in /bin working with my emulator. The last step before getting to this point was finding a bug in the 8086 CPU emulation where the pointers to things like AH and AL for the AX register were wrong so that "movb al, 1" would set ah to 1... To find that I ported ddb from FreeBSD over so I could print out registers and disassembled code that's about to execute and pore over the changes to the registers until I could spot a problem...

With that fixed, all the super simple programs run. echo(1), cat(1), tr(1), and basename(1) run. However, other simple ones like rm(1), touch(1), ls(1), and wc(1) all run into problems.

touch(1) seems to be related to a bad implementation of stat(2), for example, that I've not had the time to chase down. mv(1) and rm(1) seem that way as well. No clue what's up with wc(1) or ls(1).

I think I should come up with some kind of test script(s) to make sure the basics work.

Venix Github repo has the 86sim program in it.

I hope to soon be to the point where everything except maybe fork/exec works. I'll need those for not only /bin/sh, but also cc. cc(1) is the reason that I want to make this work so I can rebuild everything quickly for the VENIX restoration project....


Extracting part of the FreeBSD tree

Extacting the history CTM into its own

In the early days of the FreeBSD project, CTM provided a competitive advantage to the project by allowing those that weren't completely connected to the internet. CTM provided convenient way to get the sources via mag tape or mail (sometimes over UUCP links which were popular at the time).

Recently, the notion of removing from ctm from came up. Maybe it's a good idea, maybe it isn't a good idea. I thought it would be nice to know how hard it would be to extract the history of ctm into its own repo. So, I set out to see how hard it would be. Turns out, git makes it easy.

Initial setup

First, clone a tree.
git clone https://github.com/freebsd/freebsd
Next, we need to remove everything except ctm:
git filter-branch --prune-empty --subdirectory-filter usr.sbin/ctm
This will leave a repo with the complete history and a ctm directory at the top level. We're done, right?

Well, not so fast. We actually aren't done. There's a lot of crap still in the tree. Despite saying to get rid of empty commits, there are empty commits. Most of them are from merges to the tree that didn't actually touch ctm, but were merged and git has all of those in the tree.

Rebasing to clean away the sins

So, how do we clean this up? By rebasing of course. So, to make things easy, I tagged the first version in the new repo with a tag. I used 'base'. I also tagged the tip of master after the prune with 'orgmaster' in case I needed to start over (which I've not detailed here, but I'll just say it came in handy).
git rebase -i base master
Now, if you try this, it won't work. All the merges and cvs2svn 'fixups' are a problem. After a bit of trial and error, I figured out the list of things to remove from the todo list. Then I learned there was an easier way to find this list
git log --merges
will show the merges that still remain in the tree. Remove them from the todo list and then let the rebase proceed. There's likely a clever shell-script that can do all this, but I've not written one.

Sanity check

The sanity check step is easy. Diff the checked-out ctm repo with the freebsd repo's usr.sbin/ctm tree:
diff -ur ctm freebsd/usr.sbin/ctm
and it should be identical.  If not, you need to redo prior steps until it is done.

Pushing the result

So I created a new repo on github (in bsdimp/ctm).
git remote add upstream https://github.com/bsdimp/ctm
And then push it upstream:
git push upstream master
and you should be done. https://github.com/bsdimp/ctm should now have a repo that you can look at and have it match your local tree.

Next Steps

let the pull requests come in :)


Arm testing

Testing before the branch

So, tonight I tested 4 images that were 12.0 snapshots. I downloaded all the arm images that the re@ produces. The version string was 20180719-r336479.

GUMSTIX is impossible. There's no hardware it could run on, so I ignored it. It has an GUMSTIX XSCALE kernel, but a new GUMSTIX Duovero uboot and armv4 user land. Complete no go.

WANDBOARD and RPI2 booted on their hardware. I was able to dd the images and put them into my existing test bed. They just worked (but I haven't done more than just test boot, so caveat emptor).

I recently bought a new Raspberry Pi 3 B. I had a couple of RPi 3 (v1.2). I couldn't get any of them to boot at all on the RPI3 arm64 image. No output at all. [Update: thinking it may be a bad uSD, so will try again]

I also tried my old Pandaboard. It didn't work: no valid DTB was found in ubldr. [update: after selecting a good DTB, I get a core dump attaching ti_sdma_attach when calling device_get_softc()\

Later, I tried the CUBIEBOARD config on an actual Cubieboard. It worked too.

[UPDATE] PINE64 worked on my PINE64 kick-starter board. Woot!

On a related note, it was a sad day. The RPi3 B replaced my old Atmel AT91SAM9G20 board. It's my last Atmel hardware that I've turned off (I still have a ton of old Atmel gear, and some newer armv7 A5-based ones, but none of it is powered on: FreeBSD never supported the new gear, and the old gear barely works anymore).


Recovering git repo

I had a crash shortly after updating my git tree to master. When I rebooted, git was confused:
% git status
fatal: not a git repository (or any of the parent directories): .git
Needless to say, my heart skipped a beat. I have a ton of un-backed-up branches in this tree. There's many things that can cause this, according to a quick web search. However, a college told me they've seen this and the most common problem is with .git/HEAD. Looking at mine, it was 0 bytes long. This file has pointer to the latest branch checked out. Since I'd just checked out master, I was able to fix this by creating .git/HEAD:
ref: refs/heads/master
and life was good.



How to get a memory mapped serial console

Memory mapped uart

So, a friend was bringing up FreeBSD on a new system. It didn't have a traditionally mapped UART, but a memory mapped one, like we have in the embedded world. After stumbling around, I thought I'd document how to get a serial port in a situation like this.

First, he had linux running. It recognized the serial port. And it worked in UEFI.

In Linux, we first found the ttys:
% dmesg | grep tty
AMDI0020:00:ttyS4 at MMIO 0xfedc9000 (irq = 3, base_baud = 30000000) is a 16550A
AMDI0020:01:ttyS5 at MMIO 0xfedca000 (irq = 4, base_baud = 30000000) is a 16550A
ttyS4 at MMIO 0xfedc9000 (irq = 3, base_baud = 30000000) is a 16550A
ttyS5 at MMIO 0xfedca000 (irq = 4, base_baud = 30000000) is a 16550A
So, this is weird. Or rather, it's normal if you are used to ARM s16550's memory mapped, or some PCIe, PCI and CardBus serial cards.

It turns out that even if we don't have a driver for this that attaches automatically (uart works, but needs some glue on this platform), you can have a serial port console none-the-less. At the boot loader OK prompt:
OK set hw.uart.console="mm:0xfedc9000,rs:2"
What's the "rs:2"? If you look at the source it's Register Shift. It means that the cadence of registers isn't 1, but 4. It also means that each register is 32-bits, but only the lower 8 bits are valid. That's also typical in ARM and MIPS embedded processors.

So now you know.


FreeBSD Lua Loader (started in GSoC 2014) about to land

Just a quick note. I've been working on making the /stand environment easier to deal with and integrate into. After months of work on and off, the using Lua as a boot loader extension language is ready to roll into -current.

A preview review can be found here.


Setting up a SVN mirror

Setting up the svn mirror

Just following the steps in
in the Setting up a svn mirror (5.4.7) but have found there's some issues...

First step, grab the svn seed file
% fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r290116.tar.xz
this takes about 30 minutes. Also grabbed svnmirror-ports-r400410.tar.xz for a ports mirror. These file names are different than the instructions.

Next extract this somewhere (say /home/svnmirror/base) and then update it with "svnsync sync file:///home/svnmirror/base" and then set it to updating. You can do similar things for docs and ports. These files are a couple of years old now, so the sync still takes a several of hours. I believe that this is due to the protocol round trips required to extract it being sensitive to the latency between my machines and the FreeBSD svn server.

I use the following shell script as a cron job to keep things up to date.
lock="lockf -s -t 0 -k /tmp/svn-sync-mutex"

${lock} svnsync sync ${URL}/base >> ${logdir}/svn-base.log 2>&1
${lock} svnsync sync ${URL}/ports >> ${logdir}/svn-ports.log 2>&1
${lock} newsyslog -r -f ${logdir}/newsyslog.conf


FreeBSD cumulative commit graphs

FreeBSD Commit Graph

I've updated an old graph from a talk I gave in 2010 to have data up through today. The rate of development seems fairly steady.

When last I did this release, stable branches were flatter and lived longer. Now there's a fair amount of activity after the branch that continues for years.

Note: The hash marks are generally releases. I've removed the 'artificial' commits that were needed as a result of the CVS to SVN migration the project did a few years ago. The 2.0.5 branch has been omitted, and the 2.2.9 release (never tagged in the repo and < 10 changes different than 2.2.8). I had release numbers on the hash marks in earlier revs, but that's too busy now.