Fun with Foreign Debian Bootstrapping

Yesterday I found myself booting Linux on a device with no attached permanent storage - all I had was several gigabytes of RAM and the ability to netboot it through TFTP. I had been using a very minimal root filesystem inside the kernel image, but I began to wonder if it would be possible to have an entire Debian installation in the ramdisk instead - the box certainly had enough RAM to fit a minimal installation.

Ordinarily one could just use debootstrap to set up a minimal Debian installation inside a directory and make a ramdisk from that, but this was further complicated by the fact that this was a PowerPC device. Debootstrap does have a --foreign option to perform the first part of the installation on a different architecture, but the --second-stage still needs to be run as root on native hardware and assumes that it is being run from within an existing Linux installation with a bunch of standard tools available to it.

The only machines I had root on were all x86 (other than the device in question, but the ramdisk I had been using had some limitations that would have complicated matters) and some other test boxes (which I would have had to wait to requisition). So instead I decided to do a partial debootstrap on my local x86 box and complete the installation using only my local x86 box and that partial image on the PowerPC box.

If you are following this article as a guide I should note that it assumes you are able to compile and boot your own kernel and have a decent familiarity with Linux in general.

So first, begin the debootstrap process, but use --foreign to only perform the first part of the bootstrapping process (NOTE: almost everything here needs to be run as root, signified by the # at the start of each line):

# mkdir deb-ppc
# debootstrap --arch=powerpc --foreign squeeze deb-ppc http://<mirror>/debian

After this command completes you have an incomplete Debian installation in deb-ppc - some basic tools are installed (but not configured) and some packages have been downloaded but not installed. I did not select any additional packages into the initial root disk at this stage, though had I been thinking ahead it would have been useful to also include openssh-server and rsync, but that was not a major setback for me. You might want to include them, and if you don't like vi or nano you might also want to install your console editor of choice. At the moment the root disk is not bootable, so let's fix that:

# ln -s /bin/bash deb-ppc/init

This still won't boot into a full Debian installation - after the kernel finishes it's initialisation and tries to spawn the init userspace process to take over booting, it will instead spawn an interactive shell which can be used to complete the bootstrapping process. Since I'm bundling this inside the kernel image as an initramfs as opposed to an initrd loaded separately, I link an interactive shell into /init. If you were doing this with an initrd you would instead link it to /initrd.

Before we can make a ramdisk image from that directory we need to save this script as mkinitramfs.sh from Documentation/filesystems/ramfs-rootfs-initramfs.txt in the kernel sources:

#!/bin/sh

# Copyright 2006 Rob Landley <rob@landley.net> and TimeSys Corporation.
# Licensed under GPL version 2

if [ $# -ne 2 ]
then
  echo "usage: mkinitramfs directory imagename.cpio.gz"
  exit 1
fi

if [ -d "$1" ]
then
  echo "creating $2 from $1"
  (cd "$1"; find . | cpio -o -H newc | gzip) > "$2"
else
  echo "First argument must be a directory"
  exit 1
fi

NOTE: when using this script be sure you are calling this script and not a separate program also named mkinitramfs from your distribution.

Let's bundle the root disk into a cpio image:

# ./mkinitramfs.sh deb-ppc ramdisk.cpio.gz

Now you need to compile the kernel and netboot it - I'll leave the details of how to actually do that out of this article - there's plenty of good resources for that around already and the netboot procedure may vary depending on your setup (if you are netbooting at all). If you are doing this with an initramfs like I am you will need to point CONFIG_INITRAMFS_SOURCE to that image - once you have configured the kernel edit the .config file and remove the 'CONFIG_INITRAMFS_SOURCE=""' line. Then run make oldconfig which will ask you to set that option as well as some UID and GUI mapping (which you can leave as 0 since the image already should already have the correct ownership). After that you can run make and wait for the kernel to build. I'll also assume you know which zImage is the correct one to boot on your hardware.

Once you have successfully booted the kernel you should find yourself at a bash prompt. You should be aware that the environment is extremely limited at this point - for one thing there is no job control so don't try to spawn a process that you need to ctrl+c out of (I made the mistake of pinging a host to check that the network was up).

The debootstrap --second-stage did not work for me, so instead I completed the installation manually:

# export PATH=/usr/sbin:/usr/bin:/sbin:/bin
# dpkg --force-depends --install /var/cache/apt/archives/*.deb

A few things may complain during that and you may need to tell apt to fix up any problems:

# apt-get -f install

Now you will have a much more complete userspace - including vi. There's a few more things we need to do to get the system usable. Firstly, let's edit /etc/fstab and add an entry for /proc since so much userspace depends on it:

# vi /etc/fstab

proc /proc proc defaults 0 0

# mount /proc

Now we should probably get networking set up (I'm assuming you are using DHCP and your interface is eth0):

# vi /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

# vi /etc/hostname
# ifup lo
# ifup eth0

Do not make the mistake I made of checking if the interface is up by pinging something. You can run ifconfig to make sure your IP address looks right.

And set up apt (note /debian postfix in sources.list which isn't in the template provided by debootstrap - I spent around 10 minutes contemplating the 403 I was getting before I noticed that):

# vi /etc/apt/sources.list

deb http://<mirror>/debian squeeze main

# vi /etc/apt/apt.conf.d/10local

APT::Install-Recommends "0";
APT::Install-Suggests "0";

# apt-get update

Now you can install any additional packages you may need (if you didn't do this in the initial debootstrap), so let's install what we need to be able to copy our changes out of the machine (interactive SSH won't work just yet, but file copying will):

# apt-get install openssh-server rsync
# passwd

Note that if you are interacting with the machine via serial it may be a bit awkward to interact with the configuration for some packages (such as localepurge) so just install the bare essentials for the moment. After installing some packages it's probably a good idea to clean the apt cache since we are likely pretty tight on RAM:

# apt-get clean

Speaking of serial, if you are logging into the machine via serial (as I was) you may want to spawn a console on the serial line:

# vi /etc/inittab

T0:2345:respawn:/sbin/getty -L ttyS0 57600 vt100

Back on the x86 box we can now copy all those changes back into the ramdisk and make it actually boot Debian:

# rsync -avx <host>:/ deb-ppc (NOTE: the x is important, otherwise /proc will be copied as well)
# rm deb-ppc/init
# ln -s /sbin/init deb-ppc/init
# ./mkinitramfs.sh deb-ppc ramdisk.cpio.gz (again, the mkinitramfs from the kernel doc, not a distro)

Again, compile the kernel and boot it. You will need to do this last part every time you make a change in the ramdisk that you want to make persistent.

Once booted you will be able to interactively SSH into it and will find you now have a complete Debian installation you can do whatever you like with within the constraints of the available RAM. With full SSH, job control and proper TTY management you can now perform some changes that would have been a little tricky earlier, such as reconfiguring any packages you couldn't configure properly earlier (tzdata for me) and stripping out unneeded locales (this messed up a little for me since locales wasn't installed before localepurge. I haven't tested this and it's probably longer than it needs to be, but I think it will work):

# apt-get install locales
# locale-gen en_AU-UTF-8
# dpkg-reconfigure locales
# apt-get install localepurge
# localepurge
# apt-get clean

You might also want to strip out some unneeded packages, for example with:

# apt-get purge logrotate mac-fdisk rsyslog yaboot info install-info man-db manpages nano

Remember to follow the above instructions to make those changes persistent if you are happy with them. Later I'll probably play around with docpurge (from maemo) and look at other ways of reducing the size of the image (disabling logging is probably a good place to start).

If you're after some further reading on booting the kernel with initial ramdisks, check out Documentation/early-userspace/README and Documentation/filesystems/ramfs-rootfs-initramfs.txt in the kernel source.

Of Rips and Magical Musical DVDs

If there is one thing that irks me almost as much as mistagged mp3s, it's poorly encoded videos. Why is it that AVI is still so popular when Matroska containers are superior in every way? Why is MP3 still being used for the audio when any device that can play the video certainly will have enough grunt to play OGG Vorbis (codec support notwithstanding)? God forbid something encoded in ... [gasp] MPEG2 - H.264 people, H.264 (patent law notwithstanding)! Even mplayer on my Eee 701SD (running Debian Lenny) can handle all that without missing a frame!
This rant comes about as a result of me trying to buy a certain music DVD since about 5 months ago. I've had it on backorder for months, I've tried JB HiFi and some other local shops and not really trusting eBay for these kind of purchases I eventually decided to just use my left over internet quota for the month and download the thing.
On a side note, record companies - if you want to make money, why do you make it so difficult to buy things from you? "On backorder, you will receive an email when the item is back in stock", "Unfortunately xxxxx are sold out. Would you like to...", "still out of stock, there's some sorta licensing issue which is taking forever to resolve." - these are just some of the quotes I've heard as a consumer over the last year. Then there was that incident with those CDs stuck in US customs for those two months without anyone knowing where they were and costing more money as replacements were sent, then more money as they were returned after customs finally released them (So, the only thing that Free Trade Agreement did was stuff up our legal system then?)... Yeah, I'm a little off buying CDs over the Internet by now, but of course the alternative of buying in a store is quite difficult given that the bands practically have to have achieved worldwide fame to have a snowballs chance in hell of actually being in stock (ok, I am exaggerating that a little).
Now, since I live in Australia and have limits on how much I can download in a month, I strongly preference not downloading any files larger than ~700mb - and why should I? All the music DVDs I've ripped myself sound (and to a lesser extent, look) supurb at that size, surely it couldn't be that much worse than mine, right? wrong.
Now, ask yourself this - if you were ripping a music DVD, you would make sure that you set a decent bitrate on the audio track wouldn't you? I certainly would - at least 196kbps, perhaps even as high as 256kbps for those of you with ultra sensitive hearing. Well, let's just say that this particular download was a tad less than that and not go into the details too much. I won't even mention just how excruciatingly painful it was to try to listen to.
Now, it's not that hard to do a decent encoding, but it is important to have a reasonable understanding of what's actually involved in the process. It is important to know your source media - is it interlaced? Does it need to be cropped? Is there a subtitle track that you should rip as well? Is there just the one audio track; which one is the right one? Does the aspect ratio need to be fixed?
Many of those answers will vary from situation to situation and from DVD to DVD, so there isn't a one size perfectly fits all solution. Of course there are graphical tools to do this for you and some of them are no doubt pretty good, though they do not remove the need to have at least a basic understanding of what is actually happening if you want good results. I'm not going to cover any graphical tool though, I learned how to do this on the command line years ago and have stuck with that, merely expanding my knowledge when new codecs and options came out. This shell script (which I know needs work - patches welcome) is my current best practice for ripping DVDs for my personal use.
It does make a few assumptions - that the DVD is interlaced and that you want it de-interlaced (because you will be playing it on a computer monitor as opposed to a TV), that there is no subtitle track that you want to extract (if you do, add "-sid n" without the quotes and where n is the subtitle track you want, usually 0, to the end of each line starting with mencoder, though also note that there are "better" ways to do this), that this is a music DVD and not a movie (I recommend lowering the audiobitrate to 128 if it is a movie), that you only want the one default audio track (if not, specify it with mplayer's -aid option and find the appropriate ID with mplayer's -identify option), and that it doesn't need to be cropped (too error prone to automate - look at the -vf cropdetect and -vf crop options in mplayer if you need it).
You will need a few dependencies: You need the Matroska tools, Vorbis tools and x264 libraries. You will also need to make sure that you have mplayer AND mencoder built with x264 support and able to play your DVD. This probably means you will need to compile it from source, which is outside the scope of this article on account of me needing to sleep soon. Also note that depending on your location you may find that you may have legal issues regarding the patents surrounding the H.264 codec. Not to mention that you may live in a country where you cannot legally format shift or where breaking Technological Protection Measures (such as encrypted DVDs) is plain illegal - I leave it to the reader to verify that they can legally do these things or go away and complain loudly to their Government if they can't, just don't go and drag me into it all, I'm just not in the mood.

So, if you've kept reading instead of going to complain to someone in authority than I guess that you are bearing the responsibility and want to know how to actually use this.
Save it as something like rip.sh and use it like
./rip.sh filename track
where filename is the base filename you will end up with and track is the DVD track number to extract - if you leave the track blank then it will rip whatever would have played with mplayer dvd://

#!/bin/bash

targetfilesize=$[ 700 * 1024 * 1024]
audiobitrate=256

file=$1
dvddump="dvd://$2"
rawaudio="$file-rawaudio.wav"
compressedaudio="$file-compressedaudio.ogg"
pass1out="$file-pass1.avi"
pass2out="$file-pass2.avi"
finalcut="$file.mkv"

#extract audio
mplayer "$dvddump" -vc null -vo null -ao pcm:file="$rawaudio":fast </dev/null

#compress audio
oggenc "$rawaudio" -b $audiobitrate -o "$compressedaudio"
rm "$rawaudio"

#Sometimes the length of the video is misreported, so use the length of the audio track instead since it was just encoded and therefore more likely to be accurate:
#NOTE: There is a rare situation where the audio track is really not the same length as the video track - if that is the case you will need to alter this section appropriately
videolength=`echo \`mplayer -identify "$dvddump" -vo null -ao null -frames 0 2>/dev/null |awk -F= '/ID_LENGTH/ {print $2}'\` / 1 + 1 | bc`
audiolength=`echo \`mplayer -identify "$compressedaudio" -vo null -ao null -frames 0 2>/dev/null |awk -F= '/ID_LENGTH/ {print $2}'\` / 1 + 1 | bc`
echo videolength: $videolength
echo audiolength: $audiolength
length=$audiolength
echo length: $length

#calculate video bitrate
videotargetsize=$[ $targetfilesize - `du -b "$compressedaudio" | awk '{print $1}'` ]
videobitrate=`echo "$videotargetsize * 8 / $length / 1000" | bc`
echo video bitrate: $videobitrate

#video pass 1
rm divx2pass.log
mencoder "$dvddump" -vf kerndeint,scale -ovc x264 -oac lavc -lavcopts abitrate=64 -x264encopts bitrate=$videobitrate:threads=auto:pass=1:turbo=1 -o "$pass1out"

#video pass 2
mencoder "$dvddump" -vf kerndeint,scale -ovc x264 -oac lavc -lavcopts abitrate=64 -x264encopts bitrate=$videobitrate:threads=auto:pass=2 -o "$pass2out"

#compile
mkvmerge -o "$finalcut" -A "$pass2out" "$compressedaudio"

If anyone does want to submit patches for this, the main features I've been intending to implement are a more flexible command line usage, a better way to extract the audio (that doesn't have the same risk of pressing left/right yet still produces perfectly synced audio), get all the subtitle tracks embedded into the mkv file and convert the DVD chapters into a format that can be embedded into the mkv.
Update: I can't believe that I didn't think of this earlier - simply redirecting stdin from /dev/null solves the keyboard input issue when dumping the audio with mplayer.

As for me, well, I guess I'll just eBay it after all hoping it's not a bootleg and go to sleep.

Update: I'm just going to go over an issue I mentioned in this post - how to deal with media that needs it's aspect ratio corrected. The symptoms of this are generally that while you are watching a video everything just feels slightly distorted - in many cases this will be your imagination playing tricks on you, but if you are fairly certain that it isn't, read on. I'm going to use the music video for "Stick Together" which was on the bonus DVD from the album "Rock Music" by "The Superjesus" as an example. Everytime I watched this it looked distorted, so today I paused it at this frame and used the GIMP to take a screenshot:

Now, the reason I took the screenshot here is that there is a fairly large (easier to measure) drum (circular object) reasonably close to the centre of the screen (not too heavily distorted by the camera's lens) and facing the camera almost perfectly straight on (avoids perspective distortion). Using the measure tool in the GIMP I find that the drum is approximately 170 pixels wide but about 184 pixels high - clearly the aspect ratio is way out and in this case it wasn't just my imagination (phew).
You will also notice the large black bars above and below the image - these need to be cropped. And here's another reason I chose this video - take a look at this screenshot:

Notice where the black bars are and how large they are this time? This is exactly why it's important to know your source media. If I simply run mplayer -vf cropdetect over this it's going to change it's mind 3 times during playback - within the first second as that fades in it changes from crop=560:496:80:38 to crop=576:496:72:38. Then when the widescreen video starts it decides on crop=688:496:18:38. None of these are correct - the first two would cut off the left and right of the video and the last one will still leave small black bars at the top and bottom. This is one of the reasons why I mentioned that automating cropping is just too error prone. So, what's the solution? Tell mplayer to start playback after the intro artwork is gone of course! If, hypothetically, you wanted to modify my above script to attempt to detect how to crop it, I would suggest adding a line something like this (and using the crop variable in the appropriate place in the video filter chain - I cover this later):

crop=`mplayer "$dvddump" -vf cropdetect -vo null -ao null -fps 1000 -ss 60 -endpos 5|grep CROP|tail -n 1|sed 's/^.*(-vf //'|sed 's/).*$//'`

This starts the playback one minute in, quickly runs the video for 5 seconds and gives me a crop parameter of crop=688:432:18:72 - checking this with mplayer -vf crop=688:432:18:72 video.vob looks about right so it's time to move back to the problem of the aspect ratio (you could also crop the video after changing the aspect ratio - just remember to keep your video filters, including cropdetect, in the same order that you are working with).
So, let's see - I have a width of 688 pixels with the drum 170 pixels wide, and a height of 432 pixels with the drum 184 pixels high. Personally, I want to keep the width as is and scale the height to adjust the aspect. So, the currect aspect ratio is about 1.6 (688/432) and I probably want about 1.7 (432/184*170) - plugging this value into mplayer still doesn't look quite right, but I know that this is close to the standard 16:9 (1.7) aspect and a little more eyeballing tells me that's probably a bit closer. What I'm trying to get at here is that despite your best measuring efforts, it's quite difficult to get this exact and you eventually will need to just eyeball it and see if it looks good enough.
So, all together now the filter chain will look something like this:

mplayer -vf kerndeint,crop=688:432:18:72,dsize=16:9,scale=-1:-2

Breaking that down:
1. Deinterlace the video before any other processing (the absolute last thing you would ever want to do is scale first and then try to deinterlace, unless of course you like to make your eyes bleed).
2. Crop the black bars away (again, if you altered the aspect ratio before cropping the video this would be at the end of the chain).
3. dsize is used to change the intended aspect ratio used by all the following video filters (but doesn't change the aspect ratio itself).
4. Actually change the aspect ratio: a width of -1 tells it to use the original width (688 pixels), and a height of -2 tells it to scale the height using the other dimension and the intended aspect ratio.
Ok, I have lied a little - my source media is actually not interlaced in this case, so I did not use the kerndeint filter, but I wanted to drive home the point about the importance of getting the video filter order correct - I've seen it done wrong. My eyes started to bleed.

New Years Resolution: Massive Music Tag Cleanup

Once again I find that months have passed since my last entry. The blog will be a year old in little over a week and I will once again be attending linux.conf.au, this time down in Hobart. I've got myself some new gadgets - in particular a Eee PC 701SD which only cost $327 AU from JB Hifi so I have a decent computer for the conference. I'll be posting a lot more about it in the coming weeks, but am just mentioning it now as it is linked to today's post. Allow me to explain - while I've kept the default Xandros install on the internal 8 gig solid state drive I've installed Debian on a 2 gig SD card. 2 gig. yep. small, isn't it? and encrypted, but that's for another post. The point is that I've been looking for lightweight alternatives to all the software that I traditionally use in my day to day tasks, so while I'll happily leave Amarok alone on Xandros, I didn't really want to pull in all the KDE dependencies to have it on Debian, and I've come across a nice little ncurses music player called cmus to use instead.

Now, on my desktop and main laptop I use Amarok pretty much exclusively and have tried to keep all the tags in my music collection accurate - I try to check the track listing, the genre, the year and that the capitalisation complies with English capitalisation rules (except when it is apparent that the odd capitalisation is a concious decision on part of the artist and forms part of the art). I'm well aware that I've missed some - some of the artists that have been in my collection for longer still have bad capitalisation and I've only started to check the accuracy of the album years recently.

But there is a larger problem - Amarok doesn't reveal every tag to me. While that doesn't matter in the least as long as I'm only using Amarok, it can matter when I use other media players. I'm not worried about any of those albums I own a physical copy of - they're all in ogg (but if you do need a powerful ogg tag editor, tagtool's advanced mode _looks_ promising), but rather the music I've downloaded and have left in mp3. I've been aware of the issue for a while because I occasionally observe some of the symptoms on the various media players available on the Internet Tablet. I have looked at dedicated tag editors, but until now I haven't been able to find one that would show me *every* tag - not just the one's it's programmed to recognise, not just the id3v2.3 tags, but all of them. And not just the first 30 characters of them either.

Why is this *so* important, they're just extra tags, right? Well, my biggest annoyance is that cmus uses the contents of the TPE2 tag if it is present for the Artist in it's library view rather than the TPE1 tag which Amarok uses. TPE1 is defined as "Lead performer(s)/Soloist(s)", while TPE2 is defined as "Band/orchestra/accompaniment". Now, the TPE2 tag may well be perfectly valid and correct, but it is not a tag that I have been organising or validating so far with Amarok, so I'd like to get everything consistent and delete the TPE2 tags. While I'm at it, why not remove all the cover art from the mp3s - I've always felt it wasteful to keep 12 copies of the same image when I could and do just put a single image in the same folder. In fact, why not go and remove all the tags that aren't recognised by Amarok - do I really care that it was encoded with lame? I might be happy to leave the 'free download from http://www.last.fm' comment tags alone and I certainly don't want to destroy any comments that I've added, but do I really want any of the other comment tags in there?

So I finally found a id3 tag editing tool that can show me most of the tags - eyeD3. It's still not perfect - there isn't any support for id3v2.2, it doesn't show me the tags that replaygain uses and it did crash while parsing some of the mp3s - I dare say I'll have to come back to those later with another tool, even if it is hexedit. Edit: As the author pointed out, eyeD3 is in fact able to read id3v2.2 tags, just not write them and those crashes will doubtless be solved in no time.

The first step was to find out what tags are actually present in my collection:

find music -iname "*.mp3" -exec eyeD3 -v {} \; | tee index
sort -u index | awk -F\): '/^<.*$/ {print $1}' | uniq | awk -F\)\> '{print $1}' | awk -F\( '{print $(NF)}' > tags

So, that gives me a list of all the different types of tags in my collection - 44 unique tags in my case. Next step is to work out which ones are used by Amarok and if I want to keep any of the others. While I could go through and speculate on which of the three tags I can immediately see that might be a year, it's probably a better idea to look at the source code.

apt-get source amarok libtag1c2a
view amarok-1.4.9.1/amarok/src/metabundle.cpp
view taglib-1.4/taglib/mpeg/id3v2/id3v2tag.cpp

Some immediately obvious tags because it names their identifier directly are TPOS (Disc number), TBPM (beats per minute), TCOM (Composer - admittedly this is one tag that I have not been validating), TPE2 (which is marked as a non-standard MS/Apple extension - so it is aware of it but since it's messing up my collection and Amarok doesn't seem to display it anywhere I'm getting rid of it anyway) and TCMP (Compilation album, ie, show under various artists. Unfortunately cmus doesn't appear to use this tag, though does seem to have some logic for compilation albums - this is a matter I will need to investigate further later on).
Digging deeper to look past the nice friendly names that the programmers can recognise to the harsh id3 reality I also identify that I'll need to keep title (TIT2), artist (TPE1), album (TALB), comment (COMM), genre (TCON), year (TDRC) and track (TRCK) - as well as anything that is used when playing the file that isn't identified here.

Though Amarok can use images embedded in the mp3s, I don't want any - I much prefer to use Amarok's cover manager combined with copycover-offline.py to copy them into the appropriate directory (look through the comments for useful patches - hmmm, should probably submit my fix for albums with Various Artists come to think of it).

So, I made a list of these tags, one per line in a file called amaroktags. Then found all the tags in my collection that aren't supported by Amarok:

cat amaroktags tags | sort | uniq -u
view taglib-1.4/taglib/mpeg/id3v2/id3v2.4.0-frames.txt


Which left me with a list of tags that I wanted to keep:
COMM, TALB, TBPM, TCMP, TCOM, TCON, TDRC, TIT2, TPE1, TPOS, TRCK, MCDI (Music CD Identifier), TFLT (File type), TLEN (length, used for seeking), TSRC (International Standard Recording Code - the only album using it in my collection is Nine Inch Nail's Ghosts I-IV)

And an even larger list of tags to zap:
TPE2, APIC (Attached picture), TDTG (Tagging time), GEOB (arbitrary file), PCNT (Play count), POPM (Popularimeter), PRIV (private textual & binary data), TCOP (copyright), TDEN (encoding timestamp), TENC (Encoded by), TIT1 (content group description), TIT3 (Description refinement), TLAN (language), TMED (Media type), TOAL (Original title), TOFN (original filename),
TPUB (publisher), TSSE (encoding settings), TXXX (User defined text), UFID (unique file identifier), USLT (lyrics), WCOM (commercial info), WOAR (artist web page), WXXX (other URL)

As well as these ones that I couldn't identify, so I'll zap em and hope nothing breaks:
NCON, TAGC (appears to be a timestamp)

And a couple to manually check later:
TOPE (Original artist - I notice that Kong in Concert uses these for the original track names, though not accurately - they should probably be in TOAL), TYER and TDRL (years with subtly different meanings - taglib does seem to fallback and use these, but I will need to check for conflicts)

So, now I have a pretty definitive list of tags it's time to zap em' (after backing up in case something blows up in my face of course). Although not immediately obvious it appears that using the --set-text-frame specifying the 4 letter name of the frame and no contents will remove it, even if it isn't a text frame. Now, this doesn't appear to actually conserve any space in the file - it shuffles the rest of the tags upwards and zeroes out the gap (presumably conserving the space would be possible, but I don't know an easy way off the top of my head - suggestions welcome). There may be some tags that you want to have more intelligent processing on - maybe only remove some of the images or maybe only remove some of the GEOBs and if that is the case read the eyeD3 documentation, but for me I'm sick of them all and want them gone:


find music -iname "*.mp3" -exec eyeD3 --set-text-frame=TAGC: --set-text-frame=TPE2: --set-text-frame=TDTG: --set-text-frame=TCOP: --set-text-frame=TDEN: --set-text-frame=TENC: --set-text-frame=TIT1: --set-text-frame=TIT3: --set-text-frame=TLAN: --set-text-frame=TMED: --set-text-frame=TOAL: --set-text-frame=TOFN: --set-text-frame=TPUB: --set-text-frame=TSSE: --set-text-frame=TXXX: --set-text-frame=UFID: --set-text-frame=USLT: --set-text-frame=WCOM: --set-text-frame=WOAR: --set-text-frame=WXXX: --set-text-frame=NCON: --set-text-frame=APIC: --set-text-frame=GEOB: --set-text-frame=PCNT: --set-text-frame=POPM: --set-text-frame=PRIV: --set-text-frame=TCMP: {} \; | tee log


Depending on how large your collection is, at this stage you may choose to blink, stretch your arms, get some coffee, go to bed or take a vacation. Personally, I wrote a blog post.

I still have some things I know I'll have to fix up - the Deus Ex Soundtracks all seem to have multiple redundant comments, and there are some non English comment fields, but you should by this stage have a decent understanding on how to do this - that is of course, if this whole article didn't just go over your head (congrats if it did and you still read this far though :)

update: It turns out that the TCMP frame is not actually set by Amarok, so my solution is to remove all the TCMP flags from the library (I've added it to the above list, though where they are 1 in my collection is correct, but very few of the other tracks in the same album are tagged in the same way and would explain some odd behaviour when importing the albums), then to manually add them for all relevant tracks, which hopefully will ease future migration. Unfortunately as best I can tell, cmus doesn't appear to have any concept of compilation albums in it's id3.c. OGG files will supposedly get them since their tags don't require almost one thousand lines of C code to process (by contrast, cmus' vorbis.c file has a mere 285 lines including 33 lines of tag parsing), which begs the question as to why only 1 of my OGG compilation albums are marked as such in cmus.

find music/V/Various\ Artists/ -iname "*.mp3" -exec eyeD3 --set-text-frame=TCMP:1 {} \;


update: I've written a simple shell script to do this automatically, just save this as striptags.sh and execute it from your music directory:

#!/bin/sh

oktags="COMM TALB TBPM TCMP TCOM TCON TDRC TIT2 TPE1 TPOS TRCK MCDI TFLT TLEN TDTG"

indexfile=`mktemp`

#Determine tags present:
find . -iname "*.mp3" -exec eyeD3 -v {} \; > $indexfile
tagspresent=`sort -u $indexfile | awk -F\): '/^<.*$/ {print $1}' | uniq | awk -F\)\> '{print $1}' | awk -F\( '{print $(NF)}' | awk 'BEGIN {ORS=" "} {print $0}'`

rm $indexfile

#Determine tags to strip:
tostrip=`echo -n $tagspresent $oktags $oktags | awk 'BEGIN {RS=" "; ORS="\n"} {print $0}' | sort | uniq -u | awk 'BEGIN {ORS=" "} {print $0}'`

#Confirm action:
echo
echo The following tags have been found in the mp3s:
echo $tagspresent
echo These tags are to be stripped:
echo $tostrip
echo The tags will also be converted to ID3 v2.4 where appropriate
echo
echo -n Press enter to confirm, or Ctrl+C to cancel...
read dummy

#Strip 'em
stripstring=`echo $tostrip | awk 'BEGIN {FS="\n"; RS=" "} {print "--set-text-frame=" $1 ": "}'`
find . -iname "*.mp3" -exec eyeD3 --to-v2.4 $stripstring {} \; | tee -a striptags.log


Nokia to Release 3G Internet Tablet

The short version of this article is simply "I am excited". For the important part, scroll down to the bold text below, for the background, read on.

Out of *all* the gadgets that I have ever owned, the one that stands out literally miles above the rest is my Nokia N800 Internet tablet - it's well designed and can do just about anything under the sun.
My biggest (note: blowing way out of per portion for the sake of this article) problem with it has been that the only practical ways that it can access the Internet (what with being an Internet Tablet and all) is either through a wireless access point, or by using a 3G bluetooth modem. Wireless access points are quite common, but wireless access points I can legally use without annoying restrictions are comparatively rare out in the wild.
My solution to this problem was to purchase a 3G data plan (since voice + 3G data is so expensive in Australia) and take the SIM card and put it in a Nokia N95 instead of the USB modem they gave me. The advantages of this are:
- Between both devices I can do anything!
- I can get on the Internet anywhere, any-time, relatively cheap (but still not as cheap as those Americans can).
- I have a nice (for a phone) 5 mega-pixel camera that fits in my pocket, unlike my (better quality) Kodak camera.
- Not only can I use the GPS from the N95 in the N95, but I can also export it as a bluetooth GPS and use it in the mapping apps on the N800.
- In an emergency, I can use Gizmo or Skype to make up for the fact that I can't make ordinary phone calls, though thanks to the packet loss and high latency, this is not always practical.
- Should my N800 run out of power, I sometimes still have power left in one of the two batteries for the N95 and can therefore continue listening to music.
But, this set-up has disadvantages too:
- I'm always carrying around 2 devices
- I'm always carrying around a spare battery for the N95 because it often doesn't make it through the day on just one.
- The N95 needs rebooting all the time to resolve connectivity issues, especially while sharing it's Internet connection over bluetooth. It's to the point where I have NStarter installed so I can reboot faster.

Now, as I said I am very happy with the N800, and saw no reason to spend money upgrading to the N810 when it came out (although the backlit keyboard did tempt me, a lot). The one thing that would definitely make me upgrade, I said, was if Nokia added 3G support to their next Internet Tablet. Failing that I would have to take a long hard look at the specs and my money to decide.
I thought it was pretty likely that they would add 3G - it would make sense now with the iPhone out as it would put the tablet in as a direct competitor, but of course Nokia remained silent as always.

Finally, the Maemo summit arrives and I start to see an influx of posts. "The Internet Tablet line may be ending in name but the Maemo platform is going strong"? That doesn't surprise me actually. I've been speculating that their long term plans may involve Maemo ending up on their phones. Although not confirmed, it makes sense given their purchase of Trolltech and their pledge to open source Symbian - both just happen to be written in C++ and they will be able to satisfy the licence to be able to share code between them and satisfy most of the open source community at the same time.
Also, they only ever promised 5 iterations of Internet Tablets anyway, of which 4 have been released - 770, N800, N810 and N810 Wimax - though I have a feeling that they said one of those didn't count towards the 5, but I can't remember the details off hand. Whatever the future of the tablets, I think it's a safe bet that we can expect to see Maemo more and more in the future.


Now, at last I see the post I have been waiting for - Maemo 5 will have High Speed Packet Access built in - that's a 3G Tablet promised right there! They've even gone so far as to release the patches for the Linux kernel necessary to support it, so it's pretty much guaranteed now! It will also have a high definition camera, and I doubt that they would drop the GPS that the introduced in the N810, so this next tablet officially obsoletes everything I'm using my current N95 and N800 for! Well, that is of course assuming they don't ditch something else important to me, but I think that the only disadvantage will be less potential storage space upgrades.

One final point - the software the runs on the Internet Tablets is now even more Open! I don't know the full details, but the wireless drivers and and low level hardware monitoring drivers (ooh, can I fix that DSME now?) are included among the released code.

Ahh, isn't the future exciting?

Command of the Hour: Top Variants

So, I'm trying to start something that I've dubbed "Command of the Hour" on my local Canberra Linux Users Group mailing list. Quite simply, everyone just chimes in and tells everyone else about some random, obscure and useful command that they know of. Doesn't matter what, doesn't have to be related to any previous post, it just has to be something that they've found useful or can see that others might find useful.

But then I thought, why limit this to just my local LUG list? Sure it's great to test drive the idea, but why not try aiming for a wider audience - so here I am copying my initial get the ball rolling post with some top variants here:


atop - I just had an issue where gnome wasn't logging in, but seemed to be stuck constantly accessing the hard drive. This command saved me by showing me exactly which program was using the hard drive and a quick aptitude remove mlocate later my system was working perfectly again. It monitors CPU, memory, disk and network highlighting any that are particularly stressed and shows the processes responsible. Processes are only displayed if they have done something interesting
since the last update. Kernel patches can be taken to enhance the experience if one is so inclined.

htop - Awesome ncurses graphical top. Looks pretty, coloured, and simply highlighting a process and pressing 'S' will attach strace to it to see what that run away process is actually up to. Tag multiple processes and alter the niceness of them all at once or just kill em' all. 'T' toggles between process tree view and ordinary top view.

powertop - I'm sure lots of people know about this one by now, but for anyone who doesn't it can show you various information about what is chewing up energy in your system and provide some recommendations for conserving power.

iftop - top for network traffic. Shows the traffic going to and fro on every individual transfer and totalled down the bottom in ncurses bar graph style. Amounts are displayed for the last 2, 10 and 40 seconds. Filters can be applied if one is only interested in a subset of the total traffic, and it can naturally do hostname lookups and show port numbers/service names.

ntop - another network top, but this one starts a web server on port 3000 to display it's results with pretty graphs. It has the advantage that it provides much more detail - it breaks packets down by size, protocol, etc. It has many displays to analyse the data in varying and sometimes entertaining ways. Of course, being heavyweight as it is, if all you need to know is that traffic is flowing from A to B, firing this one up may be overkill, though it would easily suit as a very
quick and dirty network monitoring solution.


And a few others that I haven't found so useful myself, but someone else might:

itop - top for interrupts. I can imagine it would be useful for checking if hardware is getting the computers attention when it should be.

jnettop - this is another network top. I prefer iftop since it gives me a graphical display (and it's help page is somewhat more detailed than "I must write something here... :)").

MythTV scavanges Scrapheap Challenge episodes

My main desktop box has been running this nifty little program called MythTV for over a year now. MythTV is a home-brew Personal Video Recorder for Linux, which essentially means it's kind of like a VCR, but on steroids. It downloads TV guide data from OzTivo and once a week I go through the program guide sorted by genre to see if there is anything on that sounds like it might be worth watching. Then I check it's list of Upcoming Recordings to make sure I don't have any conflicts, resolving them if necessary and go away.
Every now and again when I have some time I bring up it's recorded programs view, pick something to watch, check that it's automatic commercial detection has done it's job making any necessary corrections (which usually takes less than a minute), sit back hit transcode (to permanently remove those commercials as well as reduce the file size) and enjoy the show.
If I'm not at my computer at the time, I can access MythTV remotely using the MythWeb plugin which lets me change my recording schedules remotely and even stream recorded programs to me. It also integrates with the MythMusic plugin which I have found handy on a number of occasions when I've been working on an assignment in a computer lab on campus and wanted some decent music to listen to ;)
Some months ago, I noticed this program in the guide called "Scrapheap Challenge" airing daily on ABC2. The description sounded interesting so I told MythTV to record it on a daily basis. Since then MythTV has not missed a single episode as all of season 9 was aired and then rolling back to the original season 1 from 1998 and showing every episode since (at the time of writing ABC2 is now up to season 7).
Each episode Scrapheap Challenge pits two teams of three members and an expert against each other to build some contraption that has to perform a specific task out of the junk they can find in a scrapheap in only 10 hours (except for some special episodes). The idea for the show came from a scene in Apollo 13 where the astronauts only had a short time to construct a carbon dioxide filter from whatever parts they could find on their space capsule. The show is hosted by Robert Llewellyn (who it took me a long time to realise played Kryten in Red Dwarf all those years ago) and Lisa Rogers (prior to season 5 it was Cathy Rogers) who offer entertaining (and sometimes ridiculous, especially in the episode introductions) commentary throughout the show.
Anyway, the reason for this post is mostly a plug for the show as I have been thoroughly enjoying it and highly recommend it to any aspiring engineer or indeed anyone with a technical mindset.

It's amazing what technology can do these days

So I was at a friends stopping the night, but unable to sleep I eventually decided to call a cab to take me home. Small dilemma: I didn't want to disturb everyone who was asleep in the house any more than I had to. But not to worry, Nokia came to my rescue!
First, I didn't know any more detail than the suburb I was in so I started Nokia Maps on my N95 (my "Mobile Modem", but that's for another post) which fairly quickly acquired a GPS fix even though I was indoors and gave me the street and even number I was at.
The web browser on the N95 is somewhat limited and although I could browse to the Canberra Cabs website with it and get their phone number, I could not use the online booking facility. Calling them would clearly disturb those sleeping around me (and I'm not even sure it would be possible given the setup I've got), so I pulled out my Nokia N800 Internet Tablet. Using my N95 as a bluetooth modem I pointed the Mozilla based browser to the Canberra Cabs website, filled in my details and within just a few minutes was in a cab spending the rest of my money for the night getting home.

Now to sleep.