Merging *BSD's usbdevs

Once upon a time, all the BSD's got their USB stack from NetBSD. With it came a usbdevs file. This file has grown and mutated on the different BSDs for some time. The time has come to merge them all back together. This sounds simple in theory, but in practice it is a lot more complicated. Different BSDs use different names for some vendors and some devices. Sometimes one is more correct than the other. Other times, they are merely both wrong. Still other times the names are the same, but the descriptions of the device are different. There's about ~2k lines in each of the files, and the merged file is closer to ~3k lines long.

I can do basically whatever I want to the FreeBSD usbdevs file to make this happen. The tricky part is justifying all the changes to a skeptical audience for the other projects. The other tricky part in merging is making sure that nothing breaks. Of course the final tricky part is getting the leg work done and the files committed quickly enough that they do not become stale. That's going to likely be the hardest part of all.

USB update

I've finished much of the work on improving the in-tree USB stack, apart from adding device IDs. This is just the initial round. Now that all the cruft has been removed from the in-tree USB stack, we'll be able to support it better in RELENG_7. I plan on merging most of the client drivers into RELENG_6 after a couple of weeks, depending on how testing goes. So far, so good. I've added about 30 new devices. In fact, I've added so many that I've had no time to update the manual pages. If there are any doc people that can help synchronize the man pages to what we now support, that would be great.

After removing the obfuscating macros from the code, it is clear to me that adding some primitive locking to the USB stack might be possible now. It isn't going to happen for 7.0, however.

ooo builds!

After the shared library version bump, the xorg 7.2 transition and the new gcc compiler in the base, I started to rebuild everything. After some snafu deleted all my +CONTENTS files, I've finally managed to rebuild everything (I think). The last major thing was ooo. After my laptop would shut down for being too hot a dozen times, and having to play whack-a-mole to remove all the extra bogus junk that's accumulated in /tmp, I've finally been able to do a build.

The biggest lesson learned, apart from how fragile java can be on amd64, is that ooo itself takes 8G of disk to build. That's /usr/ports/editors/openoffice.org-2/work only, not counting the dozens of packages it depends on. I thought I was being insanely pessimistic when I installed /tmp with 10G and put my ports tree there. Now I see that I'm right up on the edge. I'll have to repartition my disk, or get a bigger one.

Now, if only kino worked on FreeBSD/amd64 and current. It isn't even compiling for me these days :-(.

Minor USB cleanup

I did a quick pass through the tree tonight eliminating all the devinfo stuff. This code had been implemented in uhub a while ago, but the client drivers were never updated to remove the bloat. We set the device name to usbd_get_devinfo in uhub, but then in the client drivers, we computed it again, and printed it, so the data would appear twice.

So far the only wrinkle is that kldunload/kldload doesn't set the description, but there are bigger problems there.

I hope to investigate a few of the bigger usb problems before 7.0 branches and get them fixed.

CardBus and USB minor improvements

I've started working on some minor improvements to CardBus and USB. I'd like to see each of them using filters. CardBus and PC Card necessarily share interrupts between the bridge and the client devices. There's only one interrupt pin. Having to schedule an ithread just to check on the status of the card (for eject events) or power (for the interrupt driven power-up sequence) imposes a large overhead on the system. In addition, it may interfere with CardBus drivers that use filter. The current ISR isn't written to be a filter (what used to be known as a fast interrupt), so some work is needed to migrate it to such an interrupt.

For USB the problem is different. There's no interrupt sharing inside of USB, per se, since all the work goes through the pci<->usb host controller. However, on many systems the designers cheaped out and forced usb and the network to share interrupts. Since usb is still Giant locked, this causes a lot of extra Giant contention. The idea hear is to introduce a filter that turns off the interrupt bits and requests that the ISR run. The ISR would be exactly like we have today, with the addition of code to turn the interrupt sources back on. The theory here is that network inspired interrupts don't have to take out Giant at all since usb can schedule its ithread only when there's real work to do. If the ithread isn't scheduled, then Giant isn't taken out.

Since the freeze is coming up soon, I need to get this done in the next two weeks.

Found cheap SDIO 802.11b card

At CompUSA today I found a cheap 802.11b SDIO card. This card is labeled as a Kodak Camera wireless card, but has a Marvell 8686 lurking under the hood. I've seen this card before, but it was $100. Today it was $20, which was cheap enough to take a flier on. This card should come in handy when I get around to implementing an SDIO stack for the SD/MMC stack that is in FreeBSD right now. I don't know if there's any information available for this card, and I've seen indications that there might not be, but at the very least I have a card that I can parse the CIS from and do other things in preparation. OpenBSD has a partially reverse engineered driver for a PCI cousin of this card that might have some clues, but I doubt I'll find anything useful for it.

But playing with this card will have to wait until I can get the simple memory cards working with the sdh driver, or the sdmmc stuff. I need to get the ZyDas ZD1211 driver going first.

I also found a TRENDnet wireless card based on the RTL8187B chipset. Linux driver, a couple of different ones, are available. But it uses some non-standard 802.11 code, so decoding it might be interesting. It is way down in the queue at the moment, since I have no documentation for it and no known working driver. At least with the ZD1211, I have two known working drivers from which to crib.

SD/MMC

There's another SD/MMC stack for FreeBSD that's been posted. It was written by Akihiko GOTANDA-san using the SD simplified specifications. It works well for the Ricoh based parts, but fails to attach to my TI based parts. It doesn't have the hacks necessary to put the TI parts into standards conforming mode either, but even with those added there are some problems. This driver was recently announced in the mobile@ mailing list.

This joins my SD/MMC stack that I wrote for the Atmel AT91RM9200 port. Andrea Bittau also has written a 'sdh' driver that plugs into my stack. It works with Ricoh parts, but not TI parts either, but it gets closer than Gotanda-san's driver. It was announced a while ago in the mobile@ mailing list.

And there was much interest in my SD/MMC card talk at BSDcan! I'm glad to see more interest here.

ZyDas usb stick update

After my last entry, I noticed that Linux, OpenBSD and hps' new stack all have ZyDas drivers, at least for the ZD1211 and ZD1211B. hps' zyd driver appears to be a relatively recent port to his new stack.

I have the OpenBSD one builing on FreeBSD's current USB stack. I still need it for other reasons, and a MFC of HPS' stack will never happen, so if we want to support it in 6.x, we'll need one for the old stack.

I get to the part where the driver tires to setup the MAC with the RF specific register values when I get a panic. I've not had the time to debug it further, but hope to find that time soon.

My experiences with BSDcan showed that I needed both USB and CardBus wireless working...

Ativa Wireless G Network Adapter

I stopped by Office Depot today to buy a wireless usb card. I wound up getting the rather generic sounding Ativa Wireless G USB network card. This turns out to be a rebadged Belkin F5D7050 v4000 wireless card. This has the ZyDas ZD1211 chipset in it. This is a fairly popular chipset to judge by ebay (where it is just about the only chipset advertised).

There's a Linux driver. Well, there are two. One that was basically binary only that was written by ZyDas, and another that was a rewrite by the community. I'll have to see if there's one for FreeBSD :-)

eHome Cardbus card

Looks like my run of luck with CardBus cards has reached an end.

I came home with an eHome card, but it is a Marvel part for which little documentation can be found on the web. Of course, if you have info on the 88W8310 part used in this CardBus card, please let me know. This is H/W REV A1, in case they have multiple versions of the EH101.

BSDcan 2007

I had an absolute blast at BSDcan 2007.

There's more buzz this year about embedding FreeBSD than there has been in prior years. I believe that we're building momentum in this space.

There were a number of cool talks about developments in FreeBSD, which you can see at the bsdcan web site.

My talk on SD/MMC cards was well recieved. The slides are available now. This contains an overview of the various flavors of SD/MMC card.

My talk on the Atmel AT91RM9200 slides are also available, as are my 2000 talk on NEWCARD as a paper or as MagicPoint slides, and the follow on paper on ISA vs PCI interrupt dispatch in the PC Card code I gave in 2002.

I've had a chance to also hack on the cardbus resource allocation code a bit while here. There's still something screwy going on with it, but I'm having trouble locating the oddness.

I had hoped to collect some of the older SD/MMC cards while at this conference. However, nobody had any cards that I was able to take with me. I did have a few of them that I was able to try out on my Atmel AT91RM9200 board and fix minor timing issues with the stack (the older ones are a little slower and needed slightly longer timeouts).

The best part is seeing everybody again from past years. It is good to see the community growing. It is also good to see many people form companies that have been designing in FreeBSD into their products for years. These companies have also started to fund the chances necessary for them to run FreeBSD in different embedded platforms (both i386 based, as well as powerpc and mips based hardware was talked about).

There's also been some buzz about creating a FreeBSD community summit to complement the FreeBSD developers summit. A place where developers, engineers from companies using the products and users could get together and discuss how to drive FreeBSD development as well as ways to facilitate integration of changes that companies have developed for FreeBSD that they desire to give back to the FreeBSD project.

It was also good to see developers from the sister BSD projects. Renewing the personal ties to the people in these sister projects is good for the BSD ecosystem, and we need to do more of them.

That's all for now. I hope to post links to videos from my talks shortly (or find some way to put them on you tube, if they are short enough).

see everybody next year!

All in all, a very exciting BSDcan.

FreeBSD SD/MMC stack talk

My talk for BSDCan 2007 on the FreeBSD SD/MMC stack is going nicely. The basics are done, and now I'm into the hard part of the talk: actually documenting the dataflow and interfaces between layers. This is the part of the talk that will be the call to arms: more work needs to be done to add support for all the latest SD/MMC goodies.

I'll post a pointer here once I've finished the talk to my slides with notes. Maybe people here can find any silly errors in them before the question and answer session of my talk brings them up. I'll also report on the status of the Standard SD Host Adapter driver that's been written, but not yet committed to current yet. The talk also gives a lot of background for these devices, and has a number of pointers to publicly available documents that thankfully nearly totally document all aspects of SD/MMC. The exceptions being mainly the DRM features of SD and the secure MMC extensions.

FreeBSD wireless picture frame

Tonight I finished a present for my wife. A wireless picture frame based on FreeBSD.

The frame is running on an old Sony VAIO PCG-505TS that I had from many years ago. Its battery is nearly shot, good only now for powering the BIOS battery. However, I put in a solid state disk and a subset of FreeBSD. It now displays pictures from a directory on our home network in rotation.

The hardest part turned out to be how to securely mount the laptop into the picture frame.

I'll describe the software that I used in the coming days. For now, just this little teaser.

Catching up

Things at my day job are finally starting to slow down a little, so I've been trying to catch up with a bunch of changes that I made at a breakneck pace over the past several months. These should be trickling into the FreeBSD tree over the next few days and weeks.

I've setup three different Atmel boards for testing this latest batch of changes. The first board is one that my company makes. The second is a Kwikbyte KB9202A (modified to have a 32kb iic eeprom, since I burned out the first one). The third is a new eval board that I'm going to try to get running FreeBSD/arm. It has uBoot on it, so will be a departure from what I've done in the past.

I've also setup my Gateworks boards finally, so I can get FreeBSD/arm on there for testing purposes.

I also have new hardware from Atmel that should be setup soon. As well as a cute little Atmel AT91Rm9200 based smart terminal that needs to find some time to get on my dance card.

I've managed to get the necessary infrastructure into place. I have a usb hub and a bunch of serial to usb dongles to create a makeshift terminal server. I'll be transitioning this to one of the atmel boards so I can at least put it to good use.

Anyway, now that it is looking like I'll have some more time I should be writing these more often as well.

FreeBSD ExpressCard Support

I was browsing the support for ExpressCard in Linux. I couldn't find any explicit support for it at all, not matter where I looked. I found usb support and pci express support, as well as hot plug versions of the latter. This puzzled me a bit, since I'd read that Linux supported ExpressCard cards. Maybe no special software support is required at all...

So, I thought I'd get a couple of ExpressCards and take them for a test spin. It turns out that FreeBSD versions back to 4.x that can boot on modern machines with ExpressCard slots will support one important class of cards (assuming the drivers are in the version of FreeBSD that you try).

The ExpressCard standard specifies two types of cards, from an electrical point of view (not to be confused with the two width form-factors offered: 34mm and 54mm). The first kind is card that contains a USB 2.0 device in it. In this configuration, the host bridge connects it up to power and a standard usb hub. The second kind of card is a single channel PCI Express card, used for applications where greater bandwidth is required between the CPU and the hardware on the card.

The USB versions just work. I plugged in a Delkin Devices eFilm ExpressCard that I was recently able to purchase due to the generosity of a someone who sent me money to buy toys with. This device supports SD, SDHC, MMC, MS, MSpro and xD cards. Since I also wanted to enhance the SD/MMC stack to support SDHC cards, I thought this would be a good choice (since the price was right) for me to test things out on.

The PCIe versions will need PCIe hot-plug support added to pcib (or a subclass). I've had trouble finding cards that are PCIe for sure, so if you know of one, please let me know. I'm guessing that the eSATA cards are a good choice for that, but confirmation would be great.

So the good news is: FreeBSD has supported at least some of the ExpressCard cards since before the ExpressCard standard was published.

Followup Atmel AT91 family work

After receiving an evaluation board from Atmel with an AT91SAM9260 onboard, I've started updating the Atmel support for FreeBSD in the p4 repository.

This chip is basically the same as the AT91RM9200, but with a few twists. First, it uses a ARM926EJS core rather than a ARM910T core. This means that it has an arm9e core in it (half way between an arm9 and an arm10 core). Fortunately, the NetBSD support for these CPUs was easy to port, and has been nearly a drop in (I'm sure I botched something it went so smoothly).

Next, there are three classes of devices on the AT91SAM926x parts that need addressing. The first class of parts are those that have FreeBSD drivers that utilize the AT91RM9200 errata to produce proper function. Many of these bugs in silicon have been fixed, so the kernel needs to cope in some manner. The MCI device is the most prominent here, as it needed to have byte swapping done in and outbound to be interoperable with other systems. There were structural changes that severely affected performance that will need to be ifdef'd for the old part. There are a number of other minor tweaks that may be necessary.

Next, we have those devices that were present in the RM9200, but have been augmented for the SAM9 series. These include enhanced support for SDIO cards in the MCI device, as well as a number of minor additions to the TWI, SPI and USART devices (or so my first reading would suggest... some are so minor that they could just be clearer documentation).

Finally, we have those devices that aren't present at all on the RM9200, but are in the SAM9 devices. There's an LCD controller and a cool image processing whatsit that fall into this category. The EMAC from the RM9200 has been removed, replaced by a newer 10/100 part that has more buffers for transmit, so the bottlenecks seen on the RM9200's network performance can be corrected. This will take a little time, but since the SD card boot should just work, or nearly should, the pressure to have this falls from 'critical' down to 'necessary for basic support.'

The eval board I received also has some neat audio hardware attached to the SSC bus (think McASP-like bus, if you are familiar with TI's DSPs). Back of the envelope calculations suggest that I could use this device as my 'audio server'. It is small enough to slip into the rack of audio/video gear I have upstairs and fast enough to play skipless audio. This is a ways off, but time will tell.

The devices are located at different addresses than the RM9200. Initially, I'll augment the hint structure we have to cope. It should be nearly complete as it is. Eventually, however, it might be nice to have 'super kernels' that can run on any member of the Atmel family of processors that FreeBSD supports. To do that, hints will need to have some way of selecting which subset to use.

The boot loader that I carefully optimized for the RM9200 is likely too big for the SAM9. There's 16kB of SRAM on the RM9200, 12kB of which is useful for the first stage of a loader. The SAM9 has only 4kB, but a much augmented boot ROM code that may be able to help. The bootspi boot loader is about 9k with all the bells and whistles, and could be chopped to about 5k w/o much effort. I'm unsure if I should do anything here, or just have the redboot that's on the board load the kernel (or a special boot2 that runs in SDRAM rather than SRAM).

I'm sure I'm missing some other issues that will come up during the port. But so far it has been an uneventful few hours. Time, alas, conspires against me. I find I have only a few hours here or there to work on it. Although I've been working on it for weeks, I have only put 3 hours of actual grunt time into the port...

General cleanups in the newbus area

From time to time I notice things, or others do, in newbus that need to be cleaned up. These things aren't all that hard to do, usually, but finding all the places that need to do them can be a daunting task.

So I'll just report a couple that I've noticed in the past couple days, along with one from Hans Petter Selasky.

Callers of device_get_children do not check the return value. It can only fail when memory is low and the allocation fails for the children. However, it should still be checked. Going through the tree and doing sane things when it does would take some doing. This is the problem that Hans found, and many places in his usb tree he just made sure that nothing bad happened, but there might be other things necessary in some cases.

Next, not busses free up their ivars. Some just need to have a simple free(ivar) call, while others need more complicated freeing. One pass would be to audit all busses to make sure they do the right thing. This is complicated by the fact that almost all network drivers are now bus drivers too because they add mii bus to the mix.

Perhaps an idea might be to combine these two issues. device_delete_child, which would delete all the children automatically. Busses could register a bus_child_about_to_be_deleted callback which could cleanup the ivars. The default should be to just free device_get_ivar of the child deleted. Coding this up shouldn't take long, but modifying busses to use this, as well as making sure it is sufficiently general might be a bit if a challenge.

I've also done some work in p4 to create an obio concept. In this concept, all the busses in things like SoC that need help with enumeration would derive from obio (or maybe just use obio routines) which should help cut down on the large amounts of duplicated code currently present in the system. OK, small amounts, but each done almost right, but with niggling little (or large) hacks that a more regular system would solve.

I'm sure other people can suggest improvements for other problems they have noticed. Leave comments if you've seen something specific in FreeBSD and I'll update this article, or do a followup, as necessary.

Digital Camcorder to DVD on FreeBSD

A while ago, my stepson started playing competitive hockey. One of the team mothers suggested that the coach was looking for someone to tape the games and provide video. I had just purchased a camera to film the birth of my son, and there was a tripod knocking around, so I started taping the games.

I used a standard definition sony DV camera, that recorded everything in DV25 format. I then tried to transfer it to FreeBSD and create DVDs. Here's the outcome of the journey.

Ideally, there'd be one program to do this all, but there's not.

To capture the video, I used fwcontrol in the base system. I had to compile in firewire in my kernel (although the module also works). I was able to grab the video by starting fwcontrol -R, then hitting play on the camcorder. This created 10GB files.

Next, I loaded kino (from multimedia/kino) to do the editing. This is a very nice program. It handles the dv files. One word of caution. For me, sometimes it played the frames out of order, which gave me the characteristic lines on the video. mplayer and ffplay also did this. Ignore it. The final DVD will be good. Once you've edited the video, use kino's exporting function to create the mpeg files. kino will create a dvdauthor file as well. Save them both for the next step.

dvdstyler is an excellent front end to dvdauthor. The multimedia/dvdstyler port automatically installs everything it depends on. Use it to create the DVD. Most of it is fairly strait forward to use. You can even use it to edit and create menus. I needed to use the gimp to create a custom background, but those are just .jpg files. dvdstyler will either burn the DVD for you, or will let you do it. Doing it by hand is possible, but chances are excellent you'll just burn up a lot of time and wind up with an inferior set of menus...

I know this is a very high level overview. I'm blogging it so that people can find it all located in one place.

Simple hack: buy wireless card and it works

I went to OfficeMax the other day for something unreleased and noticed that they were selling 'My Essentials 802.11g wireless' cards for $40.00 each. Since I'm working on a FreeBSD-based wireless picture frame for my wife for Christmas, I thought I'd buy one and see if they worked. To my suprise, they do:

ath0: mem xxxxxxxx at device 0.0 on cardbus0
ath0: Ethernet Address 00:17:3f:xx:xx:xx
ath0: mac 7.8 phy 4.5 radio 5.6

The LEDs don't work on this card. You can enable soft led operations in the ath driver if you want the LED to function:

sysctl dev.ath.0.softled=1

More work would be needed to have the "LINK" light work from the ath driver, but not that much more. All the necessary functions appear to be exposed. The hard part may be finding the GPIO pin for the link LED, since none of the first 20 pins seems to do the trick. And after messing around with them for a while, both LEDs are on solid no matter what.

Still, for a card going into an appliance, this makes a great choice. Plus an easy hacking project if someone wanted the leds to always work.

Mustek PF-A700B Digital Photo Frame -- not hacking

Today I noticed that CompUSA has the Mustek PF-A700B digitial photo frame for $70. So I got one on a flier thinking that I could hack it.

Turns out, that I doubt I could hack it. Here's why.

The unit has two CheerTEK parts in it. A CT-950A and a CT-978. These two are the display engine and "CPU" for the box. However, there's no documentation available on these parts, or prior parts in the same family. There appears to be only 16MB RAM and no ROM/flash chips I could find.

It is kind of a shame because the box has USB, CF and SD slots. One could imagine replacing the firmware that's inside it with firmware that did a wireless connection to a network drive to get the pictures off one folder. There's also two unpopulated headers. One looks like it is either serial or JTAG. The second one is for IIC control. These could make hacking on this box relatively easy, but the total lack of documentation makes doing anything with it a non-starter.

I hope others have better luck, but I'm returning mine. It is too wimpy a photo frame, and there's no way to bend it to my needs.