Some notes on HR10-250 video

Over the past 5 or 6 years, I've had my TiVo HR10-250 setup to extract videos off of it.  Please see my other posts on how I set that up with Zipper and mfs_ftp.  There are some newer programs to help with the video extraction, but I have a cobbled together system that works OK enough for now, so I'll not go into those here.  Today I'll explore how to convert the files mfs_ftp can produce into something that looks good, at least on the TV I have :). I'm in the US, so all these instructions are NTSC-centric. Sorry for all my pals out there that have PAL.

What I will talk about is converting the tmf files (or the .ty files) into something useful. For me, useful varies with time. The tmf files are like ty files, except they are big tar balls of a series of .ty files (all smaller than some limit that I can't recall) and a xml file that describes the show. The tytompeg program is what I use to convert them to a mpg file. You can find info on how to snag this here DVRpedia entry on TyToMpg.

The tytompg program produces video that is 480x480 with a 4:3 pixel aspect ratio.  This turns out to be the SVCD resolution, so many tricks for turning SVCDs into DVDs work when processing these files. However, they aren't fully compliant SVCD files. There's some audio delay issues that cause problems and many tools have difficulties with these files. I've only overcome the synchronization problems by trial and error at this point (certain ffmpeg operations causes something to remove the stutter, sometimes on some versions). One problem with tytompg is that I lose the subtitles. I'll explore that problem more later...

A number of cartoons are broadcast these days on the SD stations in letterbox. My son watches them. It would sure by nice if I could process these files to produce a nice, 16x9 videos that play nicely. Since these files are 480x480, the actual video is in the pixels from 8,60 to 472,360 (the extra 8 pixels is a band that many stations seem to put around the picture for reasons unknown). Sometimes there's only a few pixels on each side, so it isn't worth the hassle to get rid of them (in which case the picture is bounded by 0,60 to 480x360). 720x480 is the only DVD resolution that supports a 16:9 aspect ratio. Speaking of aspect ratios, please see the writeup at http://howto-pages.org/ffmpeg/ which covers this in extreme detail. Here's the ffmpeg 1.0.1 command line I use to convert them (the 0.x command needs to crop based on the 720x480 image size).

ffmpeg -i 002-ty.mpg -vf yadif,crop=464:360:8:60,scale=720:480 -aspect 16:9 -vcodec mpeg2video -s 720x480 -b 4500k 002-nice.mpg

mpeg2video is easy to create and fairly ubiquitous, but has a couple of draw backs.  First, it is large. The above command typically doubles the size of the file. It looks really nice on the big screen HD TV (so ffmpeg's upscaler is a little better than the upscaling in the TV), but eats up the storage space. Second, many of the media renders I have in the house do mpeg2 rendering in software, so the playback can be a bit jerky on those devices.

Converting to h.264 is an option that I'll explore in later columns. The current level of tool integration isn't so nice, so would be a much longer write up.


DirecTiVo HR10-250 restore

I had to move the hard drive between two TiVos recently.  Since it was on DirecTV, I got the dreaded Error #51.  Since I couldn't recall what I did last time I got it, I googled it, only to find I needed to reset the unit with a Clear and Delete Everything (C&DE).  Turns out that since I'd hacked the TiVo, I had a much easier option that I'd forgotten about.

Sadly, I cleared everything before I'd discovered this much easier option.  I'm linking the TiVo forum post from here so that my future self will look at it before I have to go through this pain again.  Good thing I saved all the hacks I'd done to the TiVo along the way...

Recovering from Error #51

Hopefully I can easily recover the steps. Zipper sure makes it easy. It also adds the Enhancement Script that's knocking around to do all kinds of useful things. Makes hacking the TiVo way too easy...  Normally I would scoff at this script, but it is a good aggregation of knowledge that's hard to come by elsewhere, except sifting through the detritus of three or four different TiVo forums.

Anyway, just random hacking away...  Maybe the newer direct tivos can be hacked so I can download HD data again...

Oh, and one other thing: The zipper iso that's created can be booted in virtual box. If you connect your hard drive via a USB to PATA adapter, then you can connect that to the same virtual box.  The trick I found here was to create a vmdk that pointed to the drive.  When it is /dev/disk1 on my mac, I just do "VBoxManage internalcommands createrawvmdk -filename ~/TiVoDrive.vmdk -rawdisk /dev/disk1" and then connect that using the Storage tab in the config dialog for the virtual machine in VirtualBox. Handy stuff that...


FreeBSD/arm booting on SAM9260-EK to multiuser

Woot!  After working hard to get the SAM9260-EK rescued, I'm able to boot SAM9260-EK to multiuser.

Here's the dmesg:
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #13 r238762:238811M: Thu Jul 26 23:10:37 MDT 2012
    imp@dune.bsdimp.com:/dune/imp/obj/arm.arm/dune/imp/FreeBSD/sys/SAM9260EK arm
CPU: ARM926EJ-S rev 5 (ARM9EJ-S core)
  DC enabled IC enabled WB enabled LABT
  8KB/32B 4-way Instruction cache
  8KB/32B 4-way write-back-locking-C Data cache
real memory  = 67108864 (64 MB)
avail memory = 61501440 (58 MB)
at91_pmc0:  mem 0xdffffc00-0xdffffcff irq 1 on atmelarm0
at91_pmc0: Primary: 18432000 Hz PLLA: 198 MHz CPU: 198 MHz MCK: 99 MHz
at91_wdt0:  mem 0xdffffd40-0xdffffd4f irq 1 on atmelarm0
at91_wdt0: Watchdog disabled! (Boot ROM?)
at91_rst0:  mem 0xdffffd00-0xdffffd0f irq 1 on atmelarm0
at91_rst0: Reset cause: Software Request.
at91_pit0:  mem 0xdffffd30-0xdffffd39 irq 1 on atmelarm0
Timecounter "AT91SAM9 timer" frequency 6208000 Hz quality 1000
at91_pio0:  mem 0xdffff400-0xdffff5ff irq 2 on atmelarm0
at91_pio1:  mem 0xdffff600-0xdffff7ff irq 3 on atmelarm0
at91_pio2:  mem 0xdffff800-0xdffff9ff irq 4 on atmelarm0
at91_twi0:  mem 0xdffac000-0xdffaffff irq 11 on atmelarm0
iicbus0:  on at91_twi0
iic0:  on iicbus0
icee0:  at addr 0xa0 on iicbus0
at91_mci0:  mem 0xdffa8000-0xdffabfff irq 9 on atmelarm0
mmc0:  on at91_mci0
uart0:  mem 0xdffff200-0xdffff3ff irq 1 on atmelarm0
uart0: console (115200,n,8,1)
uart1:  mem 0xdffb0000-0xdffb3fff irq 6 on atmelarm0
uart2:  mem 0xdffb4000-0xdffb7fff irq 7 on atmelarm0
uart3:  mem 0xdffb8000-0xdffbbfff irq 8 on atmelarm0
uart4:  mem 0xdffd0000-0xdffd3fff irq 23 on atmelarm0
uart5:  mem 0xdffd4000-0xdffd7fff irq 24 on atmelarm0
uart6:  mem 0xdffd8000-0xdffdbfff irq 25 on atmelarm0
ate0:  mem 0xdffc4000-0xdffc7fff irq 21 on atmelarm0
miibus0:  on ate0
ukphy0:  PHY 0 on miibus0
ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ate0: Ethernet address: 3a:1f:34:08:54:54
ohci0:  mem 0xdfc00000-0xdfcfffff irq 20 on atmelarm0
usbus0 on ohci0
Timecounters tick every 10.000 msec
usbus0: 12Mbps Full Speed USB v1.0
Root mount waiting for: usbus0
ugen0.1:  at usbus0
uhub0:  on usbus0
uhub0: 2 ports with 2 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.2:  at usbus0
umass0:  on usbus0
Trying to mount root from ufs:/dev/da0s1a []...
mountroot: waiting for device /dev/da0s1a ...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0:  Removable Direct Access SCSI-0 device 
da0: 1.000MB/s transfers
da0: 483MB (990976 512 byte sectors: 64H 32S/T 483C)
warning: no time-of-day clock registered, system time will not be set accurately

I'll have to try the macb driver to see if it is any faster than the ate driver. No clue if it is working well or not.
I'll have to migrate over to booting off the NAND from the Dataflash I'm booting off of now. Having the SD card in the slot during early boot hangs because on this board both the SPI0 dataflash and the MMC are hard wired to the same, well, wires. This means I should get the nand and nandfs stuff working on this board as well, so I can boot off of its 256MB flash.
Plus I need to get back to refactoring. After editing board_sam9260ek.c I can see that I need to extend the AT91RM9200 device stuff I've done to the sam9260 (and sam9g20). But I'm not sure how far I want to go down that path. But we need some way to have a generic pin mux system. Linux has a fine example.
However, I think that many of the issues that I'm running into may be solved by moving to device-tree stuff. Other ARM ports support this now, and Linux is moving that way. Newer Atmel CPUs will have DTS files. All boards supported by Linux appear to be getting DTS files to replace the custom code for each board. It seems that this is a good way to go. AT91RM9200 is the only one that seems to not have dts files, so I may need to write those from scratch.
But the bottom line is that I'm excited. I'd given up hope on this board, and now it is booting. Woo hoo!


AT91SAM9260-EK restored!

Quite some time ago, I received a AT91SAM9260-EK board from Atmel.  I tried to turn it on at the time, and it looked dead.  I put it on the shelf and got busy with other things: new jobs, a divorce, a new marriage, a son, and a new house.

Recently, I became interested in the Atmel SoCs again (Atmel's open source group recently gave me some hardware to evaluate some work).  I took this board out of the closet and it seemed dead.  But one night I had the serial port connected and noticed that the ROM recovery program printed something.  I tried to download things with my old AT91RM9200 spi recovery program, but that didn't work.  I tried running Atmel's SAM-BA on FreeBSD, but it couldn't communicate with the board.  I tried using openocd with SAM-ICE.  Sadly, the board was wired for ICE, not JTAG and the instructions for doing the conversion online didn't work.  Finally, in an act of desperation I installed Ubuntu in as a Virtual Machine image.  With that, it worked.

So now, I have new shiny to play with that should already be supported by FreeBSD so I can make sure that I'm not breaking any at91sam9 support with my recent changed.


Wish me luck: taking the git plunge

Wish me luck.

I just typed the following commands:

  • git clone https://github.org/freebsd/freebsd-head.git
  • cd freebsd-head
  • git branch army
  • git checkout army
and started my own local branch.  Wish me luck.  Oh, I've said it, but with git, I think I might need it.  I actually didn't type exactly the above, but that's what I should have typed.  git reset seems to have been my friend.  We'll see next time I pull from github...  and if I can get the rebase command to work for me better than mercurial has been working for me at work.

Now, to figure out how to do  a similar thing to patch queues, except I really want to just correct commits rather than use a whole different set of commands.

Unifying arm boot arg parsing

Greetings again,
Here's a slightly edited version of a post I made to the FreeBSD arm list, proposing a path forward to cleanup and unify boot arg parsing on arm and at the same time add support to all boards for parsing FreeBSD's /boot/loader metadata as well as Linux's uboot/redboot meta data. Thought you might be interested.
For too long parsing boot args in arm land has suffered from cut and paste code. This is inefficient and inflexible. This patch does something to fix that. First, it modifies all the arm ports to call parse_boot_param passing in the boot parameters from initarm as the first thing in each platform's implementation of that function. This is done really super early, importantly before we start using memory outside of the loaded kernel's text, data, and bss areas. I'd thought of moving this even earlier, into __start just before the call to initarm, but wasn't completely sure was quite right (it would be more code deleted, however, if I do that: please comment). The down side is that initarm was the only function we called in __start apart from mundane things like memcpy and that would change that, but that's kinda a weak argument I think. Also, it returns the last memory location, which would be lost if I called this from __start..
I've created a weak alias to tying this function to fake_preload_metadata. All but one of the ports do this now, and this moves them to a common standard that could be more easily changed.
For most ports, it replaces the call to fake_preload_metadata. As such, I've modified the signature of fake_preload_metadata to be the same as parse_boot_param and made it a weak alias. So, if you don't define one, you'll get the current behavior.
In a future patch, I'll likely be moving the mv platform's code into this routine (I'll create a default_parse_boot_param and create a weak alias pointing to that instead). I'll need to modify the mv port to then get the dtb blob via the metadata, or possibly create a new global for it so that other platforms might make use of that also.
In a patch after that, I may add a kernel option to enable creation of FreeBSD /boot/loader metadata from Linux's standard boot stuff. This will allow platforms to get more data from the Linux boot loader without going through the intermediate /boot/loader. But it should preserve a unified interface by having it behave just like /boot/loader, but without anything setup by its more advanced features like kernel environment variables or loadable modules.
If I've done things right by this point, then any ARM port can take advantage of these new features, not just the target I'm aiming at. In addition, anybody can use their own boot loader, if they so choose, and be able to write custom code that parses the args from it in whatever appropriate way might arise for their board. I know of at least one FreeBSD/arm user that has a heavily hacked boot2 boot loader that passes things into the kernel in a non-standard way. This will accommodate them, and others like them, while still providing the project with useful functionality.
P.S. You can find the patch here.


Rearranging the deck chairs in the ARM port

Recently, I've been trying to catch up with some of the technical debt that the FreeBSD/arm port has accumulated.  Some of that technical debt was my fault, of course, but some of it wasn't.  I don't really much care whose fault things were,  but I would like to get things cleaned up.  Some of these are paths not taken.  Some are porting shims that never got properly connected.  There's also some features that were poo-poo-ed by some (including me) years ago that seem to make more sense now.

None of this stuff is terribly sexy.  Some of it will be addressed by the summer of code project that's working its way through some of the maze of twisty passages in initarm() that are subtly different between the different boards for no good reason.

We have no common place for boot loader code.  Some arm ports do use a common routine to fake up just enough metadata that debugging and other symbols work.  Some arm ports assume a full /boot/loader is present.  However, there's no common facility for getting information from via the Linux boot protocol, nor is there any provisions for having multiple boards supported by a single kernel.  There's no uniform interface to get this information that might be passed in by other means.  There's no real way for a board to get control early enough to get at meta-data a custom boot loader might pass in.

The first big thing that I'd like to comment on is a small change to the interface between locore.S' _start routine and the first port-specific code in initarm.  I've preserved the first 4 registers that were passed in on boot to _start().  Before, there'd be no way to access the Linux ATAGs, for example, by a port since these are passed in as arg 3.  To allow for future expansion, I'm just passing in one structure now.

In the coming weeks, I'll be implementing routines to parse these arguments based on standard boot protocols, while allowing customization for special needs.

In the coming months, I plan to expand our support for having multiple boards based on a single Atmel SoC.  Once I have the one at a time SoC code working, I hope to get multiple SoCs working in one kernel.  While the extra 'bloat' isn't optimal for many users, the convenience of being able to boot one kernel for installation, or trying out the system, is a feature that's been much requested.  I've already eliminated the need to have a compiled-in master clock frequency, with improved main clock detection with fallback for people using unconventional frequencies.

I'm also interested in sweeping into the tree many of the Atmel arm changes that have been accumulating in the PR database.  Some of these will be easy to integrate, while others may be difficult due to drift from the original submission.

Once we get a good baseline, I hope to merge these changes into 9.x and 8.x.  To date, the interfaces that have change have been purely internal ones we do not support.  External users may experience some bumpy waves if they haven't been working to get their changes merged upstream.  The more that you've submitted, the more I'm likely to go the extra mile to help smooth any transitions.  With luck, 8.3 and 9.2 will both benefit from these efforts.


FreeBSD Driver book

I recently received a complementary copy of the book FreeBSD Device Drivers by Josheph Kong. I also had the pleasure of meeting Josheph Kong at BSDcan 2012 a few weeks ago.
This book serves as a good introduction to all kinds of drivers and kernel modules in FreeBSD. It explores the basics of each kind of driver, complete with an example driver from the FreeBSD tree for each of the types of drivers that it describes. Joseph annotates the code to explain what each part of the code examples do, as well as to highlight the relevant areas for people wishing to write their own driver.
I'd recommend the book for anybody that needs an introduction to drivers. The walk throughs will get people up to speed, as well as introduce many of the uniquely FreeBSD quirks one needs to know to integrate with FreeBSD. The explanations are clear and concise and provide good coverage of the material.
Advanced developers will find some good nuggets in the book as well. Many of the explanations go into enough detail that even advanced users will learn things about FreeBSD from the book. Since the book sticks to the simple topics, some advanced users may be disappointed to find that some advanced topics have been omitted from this book. While many of them are very esoteric, the one I would have included would have been a discussion about how network drivers interact with struct ifnet. While other books cover that in detail for a generic BSD system, having that covered in this book would have been very beneficial. A companion book covering these topics would nicely fill the gaps here, should No Starch Press decide to publish it.
The FreeBSD world has been without any good device driver book in English for a long, long time. This book is a welcome addition. I hope it is but the first book of many from Joseph in this area.