Notes on identical .o's

The FreeBSD project just decided to remove the portability ifdefs obscuring the USB code. These ifdefs had been a growning problem for many years. They had even caused a few bugs due to subtle differences in semantics they helped to paste over. While the portability stuff may have been helpful when the code was first committed, in the long run it had the effect of making the code too difficult to maintain. The benefits from code sharing had long ago been overwhelmed by the increased maintenance burdon of its presense.

I wanted to remove this code myself. There was broad consensus that this was the right thing to do. In addition, many similar changes were being made in a USB stack rewrite that was happening outside the tree. That effort had partially removed these ifdefs, which ballooned the diffs to over 85k lines. To help with that effort as well, I thought it would make sense to make the changes in the base FreeBSD repositoy.

I wanted to make sure that any changes I made didn't break anything that was working now. Since I didn't have the resources to test every single USB device, I decided to only make those changes that didn't change the resulting .o file. If the .o file didn't change, then I knew I couldn't have broken anything. To see if a file changed, I took an MD5 of the file before and after my changes. Any change that changed the md5 I reverted. Any that didn't I retained. I thought I'd record a few of my observations and useful tricks.

First, choice of compiler flags is critical. One must have the same compiler flags between different runs. While some trivial files were the same at different levels of optimization, most of them change quite dramatically when optimization is changed.

After making dozens of changes, I hit some interesting cases where the MD5s were changing. Sometimes, when I deleted a blank line, the MD5 would change. Why would removing a blank line cause the .o to change. The answer to the puzzle turned out to be the '-g' flag used by default in the amd64 kernel config file. The key to understanding why is understanding what '-g' does. It adds meta-information about the source code to the .o file to help debuggers with their tasks. This information includes the types of the arguments to functions, the line numbers that correspond to different addresses in the .o file and the like. All of these will cause small changes to the .o, which change the MD5 produced.

Finally, after I'd removed '-g' from the command line I encountered one last issue. In FreeBSD, all the files have a $FreeBSD$ keyword. This keyword is expanded into a .comment section so that developers can find out from the binaries what sources users had when they encountered a problem. Since cvs expands this keyword to include date/time the file was committed as well as the version number of the file, committing files to the FreeBSD repository causes this to change, resulting in a change to the binary produced. One way to "sanitize" the .o files is to remove the .comment section. strip -R .comment ensures that the MD5 of the file doesn't change with the new keyword expansion after it has been committed.

Postscript

One may wonder why I used md5(1) rather than diff(1). The answer is that it is a little easier to use. "md5 *.o > /tmp/golden-md5" is easier to type and keep around than a whole lot of .o files. In addition, you still need to use the change avoidance techniques described here no matter what the method of determining "same or different."