Simple hack: buy wireless card and it works

I went to OfficeMax the other day for something unreleased and noticed that they were selling 'My Essentials 802.11g wireless' cards for $40.00 each. Since I'm working on a FreeBSD-based wireless picture frame for my wife for Christmas, I thought I'd buy one and see if they worked. To my suprise, they do:

ath0: mem xxxxxxxx at device 0.0 on cardbus0
ath0: Ethernet Address 00:17:3f:xx:xx:xx
ath0: mac 7.8 phy 4.5 radio 5.6

The LEDs don't work on this card. You can enable soft led operations in the ath driver if you want the LED to function:

sysctl dev.ath.0.softled=1

More work would be needed to have the "LINK" light work from the ath driver, but not that much more. All the necessary functions appear to be exposed. The hard part may be finding the GPIO pin for the link LED, since none of the first 20 pins seems to do the trick. And after messing around with them for a while, both LEDs are on solid no matter what.

Still, for a card going into an appliance, this makes a great choice. Plus an easy hacking project if someone wanted the leds to always work.

Mustek PF-A700B Digital Photo Frame -- not hacking

Today I noticed that CompUSA has the Mustek PF-A700B digitial photo frame for $70. So I got one on a flier thinking that I could hack it.

Turns out, that I doubt I could hack it. Here's why.

The unit has two CheerTEK parts in it. A CT-950A and a CT-978. These two are the display engine and "CPU" for the box. However, there's no documentation available on these parts, or prior parts in the same family. There appears to be only 16MB RAM and no ROM/flash chips I could find.

It is kind of a shame because the box has USB, CF and SD slots. One could imagine replacing the firmware that's inside it with firmware that did a wireless connection to a network drive to get the pictures off one folder. There's also two unpopulated headers. One looks like it is either serial or JTAG. The second one is for IIC control. These could make hacking on this box relatively easy, but the total lack of documentation makes doing anything with it a non-starter.

I hope others have better luck, but I'm returning mine. It is too wimpy a photo frame, and there's no way to bend it to my needs.

SPI and IIC into head

After discovering a number of embarassing typos, I've recently committed support for the AT91RM9200 to head for iic and spi.

The only wrinkle in the SPI integration was that i had to make it polled. When I made it use the interrupt handler, I had random corruption for reasons that are still unknown.

For the moment, the code that deals with the dataflash assumes that it is an 8MB part. This will change in the future as I get more experience with other parts.

Clonable Devices

I've seen questions on clonable devices come up from time to time. When do you want to use a clonable device? There are many times. The most common is to get 'per-open' semantics from a driver. Linux has this by default, but the structure of the vfs system in FreeBSD makes supporting it directly very difficult. Clonable devices are also useful when you want to open /dev/foo and dynamically create /dev/foo0, /dev/foo1, etc. Or when you want to have all attempts to open /dev/rodent/* succeed, yet you don't wish to enumerate all possible devices. I've heard some folks say that they want an open of /dev/tcp/client/hostname to connect to hostname, for example, and you can't enumerate all hostnames. I'll not comment on the wisdom of that design, but it is one that people understand.

Cloning it a bit underdocumented right now, as is most of the cdevsw interaction. But here's a sketch.

First, you need to register a clone event handler. There is one of these for your driver, even if it has multiple instances. The method I recommend is doing it in your attach routine if you haven't done so yet, and removing it in your detach routine:

static eventhandler_tag ehtag;
static int ehtagref;

int
foo_attach(device_t)
{
...
if (ehtag == NULL)
ehtag = EVENTHANDLER_REGISTER(dev_clone, foo_clone, 0, 1000);
ehtagref++;
...
}

int
foo_detach(device_t)
{
if (--ehtagref == 0 && ehtag != NULL)
EVENTHANDLER_DEREGISTER(dev_clone, ehtag);
}

You are then setup with a clone handler that gets called in response to all /dev/ opens that don't already have a static entry in /dev. This is a big magical, but apart from files not appearing in ls it works out well.

static struct clonedevs *foo_dev_clones = NULL;

/* Clone device */
static void
foo_dev_clone(void *arg, struct ucred *cred, char *name, int namelen,
struct cdev **dev)
{
if (*dev != NULL)
return;
/* If you want a /dev/foo control device */
if (strcmp(name, "foo") == 0)
unit = -1;
else if (dev_stdclone(name, NULL, "foo", &unit) != 1)
return; /* Bail on names we don't know */

/* find any existing device, or allocate new unit number */
if (clone_create(&foo_dev_clones, &foo_dev_cdevsw, &unit, dev, 0)) {
*dev = make_dev(&foo_dev_cdevsw, unit2minor(unit),
UID_ROOT, GID_WHEEL, 0600, "foo%d", unit);
if (*dev != NULL) {
dev_ref(*dev);
(*dev)->si_flags |= SI_CHEAPCLONE;
}
}
}

Your open routine will then get the new dev and you can save stuff off in it.


Finally, I'm looking for a good way to edit my posts so that the code comes out indented correctly. If anybody know how to do this, please let me know.

SD performance

After tweaking, I was able to get the SD performance up to 750kB/s. This was accomplished by reducing the number of commands I sent to the card, as well as upping the clock rate. I also turned off INVARIANTS and WITNESS. Here's the numbers at each step: baseline: 200kB/s. Double clock rate: 220kB/s. Remove INVARIANTS and WITNESS: 340kB/s. Remove STOP and useless select commands: 750kB/s. I think could squeeze a little more out by tightening up the code, but not more than 1 or 2kB/s.

I think I can further improve the performance to the card by in 4-bit mode. Doing multiblock reads would also help. Maybe switching from a MPSAFE interrupt to a FAST interrupt routine would improve things as well. Hard to say for sure.

However, before doing more performance tweaking, I need to make all the error paths robust in the fact of actual errors. I also need to deal with proper timeouts.

SPI Flash work

Just a quick note today.

I have most of a driver for the At91rm9200 written to deal with AT45D* (and a few other) flash parts connected via a spi bus. These parts are common boot parts, and there may be other manufacturers that have compatible parts.

In the project I'm working on, I'll need this to update the boot blocks as well as the kernel and ram disk that will live on this part. Maybe I'll update the boot blocks to cope with formatting this part as a filesystem. Given the non-standard block size, however, I'm unsure how well this will work unless I hide the non-standard size behind some kind of wear-leveling layer that takes the 1056 byte blocks and converts them to 1024 blocks. For the moment, however, I'm going to take the simple path and allow for data to be read/written and worry about these issues later.

FreeBSD MMC/SD driver update

Today was a good day.

I'm reading data from both the 256MB and 512MB SD card. In fact, I've been able to mount the root file system and run programs on them. The performance is great for a first cut, but by no means there yet. The data clock can go up to 30MHz. The Linux driver limits things to 25MHz. However, 25MHz really is 15MHz (because most people run the Atmel part with an MCK of 60MHz and you get 60 / 2 or 60 / 4 as frequencies). The Linux driver also only does 1 bit data bus. This limits them to about 2MB/s max, likely a little less. I've never tested their driver, so I don't know how fast it goes. My driver is getting about 200kB/s (at 15MHz) or 220kB/s (at 30MHz). This seems to indicate that I'm doing a lot of extra work since doubling the clock speed only gained me 10%.

Reading my driver, it looks like I'm doing a lot of extra work. For every read, we break it down into 512 byte blocks. For each block, we select the card, read a single block, issue a stop and then deselect the card. This is a lot of extra overhead. Well, it isn't QUITE that bad (we select, read 512 byte blocks, and deselect per request), but we do do the stop for every block. I'll have to see if I can eliminate most of these. I can also go to 4 bit bus, but if I do that right away I might only get 30-40kb/s. After I eliminate the extra commands, I think I'll be able to get a megabyte/s (a 5x improvement). I think I can even get more by going to multiblock commands.

I also need to implement writing to the card. And MMC support. And a bunch of other stuff.

Here are some things that I've learned.

1. When enabling the card, setting the MCIEN bit (enable) is correct, while setting MCIDIS is wrong.
2. Disabling the part with MCIDIS still allows the interrupts to happen and the device to mostly kinda work, except all replies are 0.
3. When doing data transfers, a data clock rate faster than '0' should be used to allow for the transfer to complete in a finite amount of time.
4. putting panic("oink") in unimplemented functions rather than "// XXX WRITE ME" facilitates discovery of critical, unimplemented routines.
5. Locking, but never unlocking, a lock makes it hard for other threads to acquire it.

So all in all, a good day. This should be ready to commit soon, assuming that the baby that's on the way doesn't come tomorrow... Otherwise it will be a little longer...

SD 2.0 and SDHC cards (also MMC >4GB)

I'm looking for some information and/or some hardware in this installment.

Today, I noticed that the SD 2.0 specification has been released by the SD association. The simplified version, that is. The full spec, like the 1.x versions, doesn't include the security features. Looks like there's a number of new 'host' requirements. I can't yet deside if these requirements are at a hardware or a software level yet...

So, if someone wants to purcahse me a 8GB SD or MMC card, I'll be happy to make it work.

I've started in earnest on the mmc layer. It should be done shortly. The qdmmc driver that Bernd wrote lead the way, but making commands properly be async was difficult in its structure, so I've decided to try it with the proper layering instead.

SD/MMC FreeBSD Driver Outline

There's many different flash standards in the world today. SD and MMC are two related flash card standards. These cards are used by different cameras. Readers of these cards are available for USB, PC Card and CardBus. In addition, native readers are present on many laptops and other portable devices. Here's a little background on MMC and SD. I'm writing software for MMC/SD for FreeBSD, and this will give some background for people that wish to help.

Many of the readers of MMC and SD cards emulate something else. USB cards look the same as the thumb drives, which are already supported by umass(4). PC Card and CardBus cards tend to look like ata drives, which are supported by ata(4). The native host interfaces have their own interfaces. There's two common ones in laptops, and a bunch of other interfaces in embedded devices. Since the other reads emulate something else, they aren't relevant to my FreeBSD work, so I'll not describe them further.

My efforts will concentrate on two different host adapter interfaces. The first one is the MCI interface present on the Atmel AT91RM9200 (and some newer atmel parts). This interface is relatively simple. The second one is the SD Host Controller Standard conforming host interfaces. This interface was recently documented by the SD Card Association and is present in many laptops.

For FreeBSD the software will have three layers. At the 'bottom' level is the host interface. All host interface drivers present a standard API to the bus layer. The bus layer, for the mmc/sd bus, handles all the device enumeration as well as arbitrating access to the bus. The top layer is for the SD/MMC card as well as SDIO cards that are appearing in the market place.

As a first step towards understanding, Berndt Walter write a quick and dirty driver for mmc. I added sd support for it and am debugging it. Once that's basically working, the next step is to do a proper driver with proper layering. Linux already has a system, and I'll be taking a look at it to see what I can learn from their experiences.

Anyway, back to the standards.

MMC stands for Multi Media Card. This standard was created many years ago as a small form-factor flash card. This standard was invented by the Multi Media Card Association. This standard was popular early on, but was lacking security features. The standards through version 3.31 only specifies a one wire data bus running at 25MHz. While adequate for smaller capacity cards, for large capacity cards, this presented a significant bottleneck.

The SD Association (Secure Digital) based their work on the 2.2 version of the MMC standard. They added a number of incompatible commands (on purpose, so one could tell the cards apart), changed the electrical stuff a little to make things simpler, added three more data lines as well as security features needed to deliver secure content on these cards. The SD Association later defined SDIO cards, which allow for more things than just memory cards. Today, most cards that you'll find in the marketplace today are SD cards. For a long time, this interface was totally undocumented without a nasty NDA. Recently, much of the SD standard has been released, and that's the information I'm using to create the drivers.

Cross tools and arm hacking

In the past few days I've been hacking on arm related stuff. I've worked on build tool issues (mostly integrating cogent's changes to support arm). In addition, I've had time to improve hardware support a little.

I've been getting some excellent feedback on the arm build changes, and many of them have been integrated into the tree. More to follow soon after I catch up to the changes that have already been committed which were (slightly) different than the ones I proposed. We may yet get binutils and friends merged into 6.2. The arm vs armel vs armeb issues, however, remain open.

After spending a lot of time on the build tools, it was refreshing to return to the hardware side of things. The iic bus is hanging on write transaction. I don't know if this is an interaction with the boot blocks (they too hang in the same place in some cases), or if it is a problem in the kernel code. Careful study is needed. Alas, pressures elsewhere mean that careful study must be deferred for a bit.

My main thrust has been in MMC/SD integration. I've mostly finished boot2 for at91. boot2 likely is a lame name, since it isn't the second stage boot loader, but rather the primary boot loader for this machine (redboot and uboot were rejected early on as being too large). However, despite the name (which it gets from where I stole it from), I have it loading kernels off ufs1 partitions on SD cards. I hope to get MMC card support added as well.

In addition to booting, I've been hacking on the qdmmc driver. I've added sd support, for the most part, and have refactored it to boot. Once we have this working, I'll get going on the more generic SD/MMC layer that supports things like SDIO cards.

I have a good collection of SD cards (16MB, 256MB, 512MB and 4GB), but only one MMC card (256MB). If you have MMC cards that you aren't presently using and would like to donate to my collection, please contact me at my FreeBSD email address.

Notes on identical .o's

The FreeBSD project just decided to remove the portability ifdefs obscuring the USB code. These ifdefs had been a growning problem for many years. They had even caused a few bugs due to subtle differences in semantics they helped to paste over. While the portability stuff may have been helpful when the code was first committed, in the long run it had the effect of making the code too difficult to maintain. The benefits from code sharing had long ago been overwhelmed by the increased maintenance burdon of its presense.

I wanted to remove this code myself. There was broad consensus that this was the right thing to do. In addition, many similar changes were being made in a USB stack rewrite that was happening outside the tree. That effort had partially removed these ifdefs, which ballooned the diffs to over 85k lines. To help with that effort as well, I thought it would make sense to make the changes in the base FreeBSD repositoy.

I wanted to make sure that any changes I made didn't break anything that was working now. Since I didn't have the resources to test every single USB device, I decided to only make those changes that didn't change the resulting .o file. If the .o file didn't change, then I knew I couldn't have broken anything. To see if a file changed, I took an MD5 of the file before and after my changes. Any change that changed the md5 I reverted. Any that didn't I retained. I thought I'd record a few of my observations and useful tricks.

First, choice of compiler flags is critical. One must have the same compiler flags between different runs. While some trivial files were the same at different levels of optimization, most of them change quite dramatically when optimization is changed.

After making dozens of changes, I hit some interesting cases where the MD5s were changing. Sometimes, when I deleted a blank line, the MD5 would change. Why would removing a blank line cause the .o to change. The answer to the puzzle turned out to be the '-g' flag used by default in the amd64 kernel config file. The key to understanding why is understanding what '-g' does. It adds meta-information about the source code to the .o file to help debuggers with their tasks. This information includes the types of the arguments to functions, the line numbers that correspond to different addresses in the .o file and the like. All of these will cause small changes to the .o, which change the MD5 produced.

Finally, after I'd removed '-g' from the command line I encountered one last issue. In FreeBSD, all the files have a $FreeBSD$ keyword. This keyword is expanded into a .comment section so that developers can find out from the binaries what sources users had when they encountered a problem. Since cvs expands this keyword to include date/time the file was committed as well as the version number of the file, committing files to the FreeBSD repository causes this to change, resulting in a change to the binary produced. One way to "sanitize" the .o files is to remove the .comment section. strip -R .comment ensures that the MD5 of the file doesn't change with the new keyword expansion after it has been committed.

Postscript

One may wonder why I used md5(1) rather than diff(1). The answer is that it is a little easier to use. "md5 *.o > /tmp/golden-md5" is easier to type and keep around than a whole lot of .o files. In addition, you still need to use the change avoidance techniques described here no matter what the method of determining "same or different."

Cross Building FreeBSD

I've been getting more and more questions about FreeBSD embedded systems: What is supported? How can I build for them? What can be done to improve the cross development environment? How can I help? and so on. These are all very good questions, so I thought I'd start a series of articles on the FreeBSD build system and embedding FreeBSD into products.

For this article, I'll be providing background on the FreeBSD build system as it relates to embedded systems. For an emebedded system, one usually has little more than a microcontroller with an MMU to power the entire system. While these systems work well in their target niche, they don't work so well as a general purpose platform. They lack the memory and I/O bandwidth necessary to compile applications that run on them. Server machines (or even workstations) do a much better job at these tasks, but are for a different computer architecture. One is left with the choice of compiling natively on a slow machine, or cross building on a fast machine. I'll discuss some basic aspects of cross compilation in this article.

I'll assume that you are familiar with FreeBSD's build system. If not, there are many resources available on the web. One of the better ones is the FreeBSD handbook. Future articles will build on this base, hopefully with sufficient links.

Each FreeBSD platform has a MACHINE and a MACHINE_ARCH that uniquely define it. The MACHINE_ARCH is the CPU family that executes the object code. MACHINE_ARCH is something like "i386" or "sparc64." Some CPU families come in different flavors that are mutually incompatible (such as MIPS processors using different byte ordering). These families would get a different MACHINE_ARCH for each flavor (so you'd have mipseb and mipsel, for example, to distinguish the two different flavors of MIPS CPUs). If you are familar with the gnu compilers and such, this field roughly corresponds to "CPU" field of their "CPU-MANUFACTURER-OS" identifier. I say "corresponds" because FreeBSD uses "amd64" as the MACHINE_ARCH for AMD's extensions to the x86 architecture (the official name of the architecture), while gcc uses the older x86_64 designation.

MACHINE defines the way the machine is put together. Often times machines are build with the same CPU, but with differing glue chips and support software. In cases like this, if the machines are very similar, FreeBSD will support that with one MACHINE type. In other cases, the differences are large enough to warrant an entirely separate MACHINE for that machine. What differentiates one MACHINE type from another with the same MACHINE_ARCH are things like BIOS interface, boot procedure, disk partitioning, and expansion bus technologies. For example, the FreeBSD/i386 and FreeBSD/pc98 platforms both execute on i386 compatible CPUs, but they have completely different bus topologies, boot procedures, BIOS interfaces, and disk layout.

When one builds FreeBSD on one platform to execute on another platform, that is called cross compiling. Cross compilation support is integrated into the normal FreeBSD build system. It is activated by defining TARGET and TARGET_ARCH of the machine you are targeting and using the normal make targets. For example:

make TARGET=arm TARGET_ARCH=arm buildworld

will build the entire userland portion of the system for an ARM machine. Similarly, the following:

make TARGET=arm TARGET_ARCH=arm installworld DESTDIR=/foo

will install these components into the directory tree /foo. One can even build and install a kernel for the target platform:

make TARGET=arm TARGET_ARCH=arm KERNCONF=BONGO kernel DESTDIR=/foo

After cross compilation support was added to the tree, the FreeBSD project added two additional make targets which allow use of the build tools outside of the build tree. The first is "buildenv." It forks a shell with the same build environment used to build FreeBSD. This target is for interactive development. Much of the Atmel AT91RM9200 port was done using this target, for example. However, there's no ability to specify a command to run in this sub-shell, nor does it lend itself well to automated building systems. The make target "buildenvvar" was created to try to fill the gap in the latter problem. This target prints the build environment variables used by the build system. We used it at my day joy to help automate building our products on an x86 processor for an embedded ARM design we're producing. This works well for many applications, but not for all. We have found that we can make it work for simple applications, but more complicated one, with more elaberate build systems, require something more sophisticated. In a future article, I'll describe how we solved that problem.

Many times, cross building can be useful for "compile testing" changes to the tree. There's a make target called "universe" that builds the entire tree for each of the Tier 1 and 2 architectures, along with all the kernels that it can find. In many cases, these "compile tests" of the system find problems that would otherwise be found by others, thus saving much embarassment. The drawback, however, to a "universe" build is that it takes several hours on a very fast machine.

I hope that you've found the above information useful, and that it answers your basic questions in this area. Feel free to leave feedback with additional questions or requests for clarification and I'll see what I can do about addressing them in the feedback section, or in future articles.

FreeBSD Transparent Web Proxie with Squid

Introduction

Over the past few days, I've been setting up a transparent proxie. I have a need to limit traffic to a small set of sites for a while, so I thought I'd look into setting up a transparent proxie with squid.

I run nothing but FreeBSD machines for my home infrastructure where possible. I easily found on the web a number of web sites that talked about using cisco routers to FreeBSD backends to create enterprise level web filters for corporations, or variations on this theme. These articles served to wet my appetite only, but didn't fulfill my needs. I don't have a cisco router, just a dinky soekris box running NanoBSD build from FreeBSD 6.1-RELEASE with a 6.1-STABLE kernel as of a few days ago. Many of the other articles discussed using pf and squid together, but these required that pf and squid run on the same machine. While the router could easily run pf, running squid was likely to prove impossible.

I was given a solution by a friend on IRC that I'll share here now so that future folks wishing to implement this in the future can find it via a web search.

I used ipfw redirection + squid to implement my transparent cache/access control mechanism. The router and the beefy squid box were on the same L2 network (same ethernet network, no routers) so I was able to use the forward function of ipfw to forward the web packets to the squid machine. The squid machine then redirected this traffic to the squid cache port, which then served up the pages that were requested.

Wimpy SOEKRIS box

On my Soekris box, I added the following lines to my kernel:

# Firewall stuff
options IPFIREWALL
options IPFIREWALL_FORWARD

Once I had the firewall stuff in my kernel, I had to write the firewall rules. Thinking that this would be easy after I rebooted, I learned again that ipfw defaults to deny, so I had to get out my serial console to correct this problem. I'd recommend getting the system completely debugged using a kernel with "options IPFIREWALL_DEFAULT_TO_ACCEPT" and then removing it after you are sure you don't need it.

After arranging for a serial console, I was able to write the firewall rules I needed. Since I have multiple firewalls protecting my network, the rules I used were relatively simple. The soekris router was acting as a router with a little bit if filtering as a backstop to the linux-based DSL modem that I have, which has some rudementary forwarding builtin, but it never hurts to have belts and suspenders. Here's the relevant rulesets that I used:

/sbin/ipfw add 1000 pass tcp from 10.11.13.80 to any
/sbin/ipfw add 1100 fwd 10.11.13.80 tcp from 10.11.13.0/24 to any 80
/sbin/ipfw add 65000 pass all from any to any

The first rule is to allow my beefy squid host (10.11.13.80) to get to the outside world for squid's network requests. It is very important that this rule be listed first so that the second rule doesn't cause an infinite loop.

The second rule redirects all web traffic (well, all traffic to port 80) to the beefy squid host.

The last rule is there to make sure that all other traffic is allowed. These rules will likely be part of a more compled rule set. The first few should be near the top, after all the sanity checks for obviously spoofed packets have been done. The last one, if you choose to have it, should be near the end of the list. If you've taken the time to implement a complete list of what's allowed, then change 'pass' to 'deny'.

Once you can see the packets on the beefy squid host with tcpdump, you are ready to configure that machine. But before we go onto that, here's the start of the dmesg to show that the Soekris box is really a small box. 64MB ram, with a 133MHz AMD Elan CPU:

Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-STABLE #2: Mon Aug 21 00:32:38 MDT 2006
imp@paco-paco.bsdimp.com:/tmp/obj/i386/pe/imp/FreeBSD/6x/src/sys/SOEKRIS
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Enhanced Am486DX4/Am5x86 Write-Back (486-class CPU)
Origin = "AuthenticAMD" Id = 0x494 Stepping = 4
real memory = 67108864 (64 MB)
avail memory = 60272640 (57 MB)

I also needed to setup /etc/rc.conf so that the firewall would be enabled:

firewall_enable=YES
firewall_script=/etc/rc.firewall.router

The firewall script I had earlier I placed in the /etc/rc.firewall.router file.

Beefy Squid Machine

My beefy squid machine was a Dell box with a lot of memory and a fast 3.0GHz Intel EMT64 dual core:

Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 7.0-CURRENT #1: Mon Aug 21 19:36:37 MDT 2006
imp@paco-paco.bsdimp.com:/pe/imp/p4/arm/src/sys/amd64/compile/PACO
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) D CPU 3.00GHz (3000.12-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0xf47 Stepping = 7
Cores per package: 2
usable memory = 4282433536 (4084 MB)
avail memory = 4137197568 (3945 MB)

This machine ran squid and ipfw as well. ipfw needed to have the following kernel options to make it work:

options IPFIREWALL
options IPFIREWALL_FORWARD
options IPFIREWALL_VERBOSE

Note: If this machine had been a 5.x or 6.x machine, you'd also need 'options IPFIREWALL_FORWARD_EXTENDED' for this to work. 4.x and 7.0 machines won't need this extra option.

Again, I needed to write firewall rules. For this machine, I needed to redirect all that web traffic to squid. Here's what I wrote:

# allow this machine to go to the net unmolested for port 80 traffic
/sbin/ipfw add 900 pass all from 10.11.13.80 to any 80
# all other traffic goes to squid
/sbin/ipfw add 1000 log fwd 127.0.0.1,3128 tcp from 10.11.13.0/24 to any 80
# everything else is cool
/sbin/ipfw add 65000 pass all from any to any

Since this was an internal machine, I didn't see the harm in passing all data. Your milage may vary.

I needed to configure squid. well, first I needed to install squid, but I just built it using the FreeBSD squid port found in /usr/ports/www/squid. I found some online resources here, and came up with:

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl ten_net src 10.11.13.0/24
http_access allow ten_net
http_access deny all
http_reply_access allow all
icp_access allow all
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
coredump_dir /usr/local/squid/cache

The above config is mostly the stock, plus the 2.5 and earlier transparent cacheing recipe that can be found on many other web sites.

I also needed to add the following to /etc/rc.conf:

squid_enable=YES
firewall_enable=YES
firewall_script=/etc/rc.firewall.paco

Blocking one website

Now that I have all of the above configured, I can proceed to blocking one web site that I started to in the first place. That turns out to be relatively easy. I just added the following to squid.conf:

acl dst_deny dstdomain .foo.com
deny_info ERR_NO_FOO dst_deny
http_access deny dst_deny

I also had to add a file /usr/local/etc/squid/errors/English/ERR_NO_FOO with a custom message and reminder about acceptable use. I copied the file ERR_TOO_BIG and hacked the text so it looked OK.

Also note: this block was put in place for a few days to serve as logical consequences for abusing the foo.com priviledge by a minor that has access to the network. If is unknown how well these sorts of blocks will work in the long run, and the author believes that they are of limited use for limited circumstances. Ideally, one would be able to trust completely everybody on the network, but that's not always possible. This might be a useful tool, but it is more to keep honest people honest. There's a number of proxies and such that can be employed to evade this sort of policing, etc.

Conclusion

As you can see, setting up this proxie isn't too bad. Once you have it in place, you can add additional bells and whistles as described in the squid FAQ and other documents. You can filter/cache more traffic. You can even run some of the add blocking software that is available as a squid add-on.

Thecus N4100

spent much of yesterday trying to break into the Thecus N4100 that we have at work. I thought it would be a relatively easy thing to try. After all, OpenBSD supports the N2100 and how different can this one be? thought it might be nice to add FreeBSD support to the thing.

Turns out, quite a lot.

I was greeted with gibberish over the serial console port. 115200 looks like the correct baud rate, at least based on looking at the characters with a scope. The bit widths are right, etc. Yet all I get is garbage.

Then I went and googled for others that have done this. While $600 for the empty unit is a bunch of money, it isn't that much in the grand scheme of things. I was surprised to find that no one has a web page about this that google can get to. Maybe I need to learn to search better? Or maybe there's limited searching options for this community.

I've tried jumpers on all the things that looked like jumpers. I've tried every combination possible, but still no joy on the serial port.

The unit works, and is running Thecus' software. The GPL files showed no signs of what the key might be to the serial communications. The settings were 115200, 8, N. Maybe those settings are dependent on a clock that's off by more than a few percent, which is why the serial ports I've tried can't lock to it. I'll have to give the embedded ATMEL part that I'm using a spin to see if I can vary the baud rate enough to see it (the embedded part I use has a base xtal that all frequencies are based on, so I've seen my share of non-standard baud rates debugging it as I went from board to board that had different frequencies).

I'm also putting this out as a kind of test to see if I can find people who have done this sort of thing, or would like to. I'm not sure of how well Google covers blogs, so now I have something to search on.

In the future, I'll try to see if I can kick this CPU into a 'useful' recovery mode so I can just take over from scratch before redboot even gets around to starting :-0

How to recover a hard disk

OK. I'd like to spend some time and find out how hard it is to do DIY data recovery from a hard disk drive. I had one crash and burn while I was at BSDCan this year. Now that the move is done, I'll have time again to take it apart, and put it back together 'fixed'. The drive motor is shot, so I guess I'll start by going onto EBay and buying one.

Wish me luck!