Friday, 16 January 2009

New Years Resolution: Massive Music Tag Cleanup

Once again I find that months have passed since my last entry. The blog will be a year old in little over a week and I will once again be attending linux.conf.au, this time down in Hobart. I've got myself some new gadgets - in particular a Eee PC 701SD which only cost $327 AU from JB Hifi so I have a decent computer for the conference. I'll be posting a lot more about it in the coming weeks, but am just mentioning it now as it is linked to today's post. Allow me to explain - while I've kept the default Xandros install on the internal 8 gig solid state drive I've installed Debian on a 2 gig SD card. 2 gig. yep. small, isn't it? and encrypted, but that's for another post. The point is that I've been looking for lightweight alternatives to all the software that I traditionally use in my day to day tasks, so while I'll happily leave Amarok alone on Xandros, I didn't really want to pull in all the KDE dependencies to have it on Debian, and I've come across a nice little ncurses music player called cmus to use instead.

Now, on my desktop and main laptop I use Amarok pretty much exclusively and have tried to keep all the tags in my music collection accurate - I try to check the track listing, the genre, the year and that the capitalisation complies with English capitalisation rules (except when it is apparent that the odd capitalisation is a concious decision on part of the artist and forms part of the art). I'm well aware that I've missed some - some of the artists that have been in my collection for longer still have bad capitalisation and I've only started to check the accuracy of the album years recently.

But there is a larger problem - Amarok doesn't reveal every tag to me. While that doesn't matter in the least as long as I'm only using Amarok, it can matter when I use other media players. I'm not worried about any of those albums I own a physical copy of - they're all in ogg (but if you do need a powerful ogg tag editor, tagtool's advanced mode _looks_ promising), but rather the music I've downloaded and have left in mp3. I've been aware of the issue for a while because I occasionally observe some of the symptoms on the various media players available on the Internet Tablet. I have looked at dedicated tag editors, but until now I haven't been able to find one that would show me *every* tag - not just the one's it's programmed to recognise, not just the id3v2.3 tags, but all of them. And not just the first 30 characters of them either.

Why is this *so* important, they're just extra tags, right? Well, my biggest annoyance is that cmus uses the contents of the TPE2 tag if it is present for the Artist in it's library view rather than the TPE1 tag which Amarok uses. TPE1 is defined as "Lead performer(s)/Soloist(s)", while TPE2 is defined as "Band/orchestra/accompaniment". Now, the TPE2 tag may well be perfectly valid and correct, but it is not a tag that I have been organising or validating so far with Amarok, so I'd like to get everything consistent and delete the TPE2 tags. While I'm at it, why not remove all the cover art from the mp3s - I've always felt it wasteful to keep 12 copies of the same image when I could and do just put a single image in the same folder. In fact, why not go and remove all the tags that aren't recognised by Amarok - do I really care that it was encoded with lame? I might be happy to leave the 'free download from http://www.last.fm' comment tags alone and I certainly don't want to destroy any comments that I've added, but do I really want any of the other comment tags in there?

So I finally found a id3 tag editing tool that can show me most of the tags - eyeD3. It's still not perfect - there isn't any support for id3v2.2, it doesn't show me the tags that replaygain uses and it did crash while parsing some of the mp3s - I dare say I'll have to come back to those later with another tool, even if it is hexedit. Edit: As the author pointed out, eyeD3 is in fact able to read id3v2.2 tags, just not write them and those crashes will doubtless be solved in no time.

The first step was to find out what tags are actually present in my collection:

find music -iname "*.mp3" -exec eyeD3 -v {} \; | tee index
sort -u index | awk -F\): '/^<.*$/ {print $1}' | uniq | awk -F\)\> '{print $1}' | awk -F\( '{print $(NF)}' > tags

So, that gives me a list of all the different types of tags in my collection - 44 unique tags in my case. Next step is to work out which ones are used by Amarok and if I want to keep any of the others. While I could go through and speculate on which of the three tags I can immediately see that might be a year, it's probably a better idea to look at the source code.

apt-get source amarok libtag1c2a
view amarok-1.4.9.1/amarok/src/metabundle.cpp
view taglib-1.4/taglib/mpeg/id3v2/id3v2tag.cpp

Some immediately obvious tags because it names their identifier directly are TPOS (Disc number), TBPM (beats per minute), TCOM (Composer - admittedly this is one tag that I have not been validating), TPE2 (which is marked as a non-standard MS/Apple extension - so it is aware of it but since it's messing up my collection and Amarok doesn't seem to display it anywhere I'm getting rid of it anyway) and TCMP (Compilation album, ie, show under various artists. Unfortunately cmus doesn't appear to use this tag, though does seem to have some logic for compilation albums - this is a matter I will need to investigate further later on).
Digging deeper to look past the nice friendly names that the programmers can recognise to the harsh id3 reality I also identify that I'll need to keep title (TIT2), artist (TPE1), album (TALB), comment (COMM), genre (TCON), year (TDRC) and track (TRCK) - as well as anything that is used when playing the file that isn't identified here.

Though Amarok can use images embedded in the mp3s, I don't want any - I much prefer to use Amarok's cover manager combined with copycover-offline.py to copy them into the appropriate directory (look through the comments for useful patches - hmmm, should probably submit my fix for albums with Various Artists come to think of it).

So, I made a list of these tags, one per line in a file called amaroktags. Then found all the tags in my collection that aren't supported by Amarok:

cat amaroktags tags | sort | uniq -u
view taglib-1.4/taglib/mpeg/id3v2/id3v2.4.0-frames.txt


Which left me with a list of tags that I wanted to keep:
COMM, TALB, TBPM, TCMP, TCOM, TCON, TDRC, TIT2, TPE1, TPOS, TRCK, MCDI (Music CD Identifier), TFLT (File type), TLEN (length, used for seeking), TSRC (International Standard Recording Code - the only album using it in my collection is Nine Inch Nail's Ghosts I-IV)

And an even larger list of tags to zap:
TPE2, APIC (Attached picture), TDTG (Tagging time), GEOB (arbitrary file), PCNT (Play count), POPM (Popularimeter), PRIV (private textual & binary data), TCOP (copyright), TDEN (encoding timestamp), TENC (Encoded by), TIT1 (content group description), TIT3 (Description refinement), TLAN (language), TMED (Media type), TOAL (Original title), TOFN (original filename),
TPUB (publisher), TSSE (encoding settings), TXXX (User defined text), UFID (unique file identifier), USLT (lyrics), WCOM (commercial info), WOAR (artist web page), WXXX (other URL)

As well as these ones that I couldn't identify, so I'll zap em and hope nothing breaks:
NCON, TAGC (appears to be a timestamp)

And a couple to manually check later:
TOPE (Original artist - I notice that Kong in Concert uses these for the original track names, though not accurately - they should probably be in TOAL), TYER and TDRL (years with subtly different meanings - taglib does seem to fallback and use these, but I will need to check for conflicts)

So, now I have a pretty definitive list of tags it's time to zap em' (after backing up in case something blows up in my face of course). Although not immediately obvious it appears that using the --set-text-frame specifying the 4 letter name of the frame and no contents will remove it, even if it isn't a text frame. Now, this doesn't appear to actually conserve any space in the file - it shuffles the rest of the tags upwards and zeroes out the gap (presumably conserving the space would be possible, but I don't know an easy way off the top of my head - suggestions welcome). There may be some tags that you want to have more intelligent processing on - maybe only remove some of the images or maybe only remove some of the GEOBs and if that is the case read the eyeD3 documentation, but for me I'm sick of them all and want them gone:


find music -iname "*.mp3" -exec eyeD3 --set-text-frame=TAGC: --set-text-frame=TPE2: --set-text-frame=TDTG: --set-text-frame=TCOP: --set-text-frame=TDEN: --set-text-frame=TENC: --set-text-frame=TIT1: --set-text-frame=TIT3: --set-text-frame=TLAN: --set-text-frame=TMED: --set-text-frame=TOAL: --set-text-frame=TOFN: --set-text-frame=TPUB: --set-text-frame=TSSE: --set-text-frame=TXXX: --set-text-frame=UFID: --set-text-frame=USLT: --set-text-frame=WCOM: --set-text-frame=WOAR: --set-text-frame=WXXX: --set-text-frame=NCON: --set-text-frame=APIC: --set-text-frame=GEOB: --set-text-frame=PCNT: --set-text-frame=POPM: --set-text-frame=PRIV: --set-text-frame=TCMP: {} \; | tee log


Depending on how large your collection is, at this stage you may choose to blink, stretch your arms, get some coffee, go to bed or take a vacation. Personally, I wrote a blog post.

I still have some things I know I'll have to fix up - the Deus Ex Soundtracks all seem to have multiple redundant comments, and there are some non English comment fields, but you should by this stage have a decent understanding on how to do this - that is of course, if this whole article didn't just go over your head (congrats if it did and you still read this far though :)

update: It turns out that the TCMP frame is not actually set by Amarok, so my solution is to remove all the TCMP flags from the library (I've added it to the above list, though where they are 1 in my collection is correct, but very few of the other tracks in the same album are tagged in the same way and would explain some odd behaviour when importing the albums), then to manually add them for all relevant tracks, which hopefully will ease future migration. Unfortunately as best I can tell, cmus doesn't appear to have any concept of compilation albums in it's id3.c. OGG files will supposedly get them since their tags don't require almost one thousand lines of C code to process (by contrast, cmus' vorbis.c file has a mere 285 lines including 33 lines of tag parsing), which begs the question as to why only 1 of my OGG compilation albums are marked as such in cmus.

find music/V/Various\ Artists/ -iname "*.mp3" -exec eyeD3 --set-text-frame=TCMP:1 {} \;


update: I've written a simple shell script to do this automatically, just save this as striptags.sh and execute it from your music directory:

#!/bin/sh

oktags="COMM TALB TBPM TCMP TCOM TCON TDRC TIT2 TPE1 TPOS TRCK MCDI TFLT TLEN TDTG"

indexfile=`mktemp`

#Determine tags present:
find . -iname "*.mp3" -exec eyeD3 -v {} \; > $indexfile
tagspresent=`sort -u $indexfile | awk -F\): '/^<.*$/ {print $1}' | uniq | awk -F\)\> '{print $1}' | awk -F\( '{print $(NF)}' | awk 'BEGIN {ORS=" "} {print $0}'`

rm $indexfile

#Determine tags to strip:
tostrip=`echo -n $tagspresent $oktags $oktags | awk 'BEGIN {RS=" "; ORS="\n"} {print $0}' | sort | uniq -u | awk 'BEGIN {ORS=" "} {print $0}'`

#Confirm action:
echo
echo The following tags have been found in the mp3s:
echo $tagspresent
echo These tags are to be stripped:
echo $tostrip
echo The tags will also be converted to ID3 v2.4 where appropriate
echo
echo -n Press enter to confirm, or Ctrl+C to cancel...
read dummy

#Strip 'em
stripstring=`echo $tostrip | awk 'BEGIN {FS="\n"; RS=" "} {print "--set-text-frame=" $1 ": "}'`
find . -iname "*.mp3" -exec eyeD3 --to-v2.4 $stripstring {} \; | tee -a striptags.log

Friday, 19 September 2008

Nokia to Release 3G Internet Tablet

The short version of this article is simply "I am excited". For the important part, scroll down to the bold text below, for the background, read on.

Out of *all* the gadgets that I have ever owned, the one that stands out literally miles above the rest is my Nokia N800 Internet tablet - it's well designed and can do just about anything under the sun.
My biggest (note: blowing way out of per portion for the sake of this article) problem with it has been that the only practical ways that it can access the Internet (what with being an Internet Tablet and all) is either through a wireless access point, or by using a 3G bluetooth modem. Wireless access points are quite common, but wireless access points I can legally use without annoying restrictions are comparatively rare out in the wild.
My solution to this problem was to purchase a 3G data plan (since voice + 3G data is so expensive in Australia) and take the SIM card and put it in a Nokia N95 instead of the USB modem they gave me. The advantages of this are:
- Between both devices I can do anything!
- I can get on the Internet anywhere, any-time, relatively cheap (but still not as cheap as those Americans can).
- I have a nice (for a phone) 5 mega-pixel camera that fits in my pocket, unlike my (better quality) Kodak camera.
- Not only can I use the GPS from the N95 in the N95, but I can also export it as a bluetooth GPS and use it in the mapping apps on the N800.
- In an emergency, I can use Gizmo or Skype to make up for the fact that I can't make ordinary phone calls, though thanks to the packet loss and high latency, this is not always practical.
- Should my N800 run out of power, I sometimes still have power left in one of the two batteries for the N95 and can therefore continue listening to music.
But, this set-up has disadvantages too:
- I'm always carrying around 2 devices
- I'm always carrying around a spare battery for the N95 because it often doesn't make it through the day on just one.
- The N95 needs rebooting all the time to resolve connectivity issues, especially while sharing it's Internet connection over bluetooth. It's to the point where I have NStarter installed so I can reboot faster.

Now, as I said I am very happy with the N800, and saw no reason to spend money upgrading to the N810 when it came out (although the backlit keyboard did tempt me, a lot). The one thing that would definitely make me upgrade, I said, was if Nokia added 3G support to their next Internet Tablet. Failing that I would have to take a long hard look at the specs and my money to decide.
I thought it was pretty likely that they would add 3G - it would make sense now with the iPhone out as it would put the tablet in as a direct competitor, but of course Nokia remained silent as always.

Finally, the Maemo summit arrives and I start to see an influx of posts. "The Internet Tablet line may be ending in name but the Maemo platform is going strong"? That doesn't surprise me actually. I've been speculating that their long term plans may involve Maemo ending up on their phones. Although not confirmed, it makes sense given their purchase of Trolltech and their pledge to open source Symbian - both just happen to be written in C++ and they will be able to satisfy the licence to be able to share code between them and satisfy most of the open source community at the same time.
Also, they only ever promised 5 iterations of Internet Tablets anyway, of which 4 have been released - 770, N800, N810 and N810 Wimax - though I have a feeling that they said one of those didn't count towards the 5, but I can't remember the details off hand. Whatever the future of the tablets, I think it's a safe bet that we can expect to see Maemo more and more in the future.


Now, at last I see the post I have been waiting for - Maemo 5 will have High Speed Packet Access built in - that's a 3G Tablet promised right there! They've even gone so far as to release the patches for the Linux kernel necessary to support it, so it's pretty much guaranteed now! It will also have a high definition camera, and I doubt that they would drop the GPS that the introduced in the N810, so this next tablet officially obsoletes everything I'm using my current N95 and N800 for! Well, that is of course assuming they don't ditch something else important to me, but I think that the only disadvantage will be less potential storage space upgrades.

One final point - the software the runs on the Internet Tablets is now even more Open! I don't know the full details, but the wireless drivers and and low level hardware monitoring drivers (ooh, can I fix that DSME now?) are included among the released code.

Ahh, isn't the future exciting?

Tuesday, 16 September 2008

Command of the Hour: Top Variants

So, I'm trying to start something that I've dubbed "Command of the Hour" on my local Canberra Linux Users Group mailing list. Quite simply, everyone just chimes in and tells everyone else about some random, obscure and useful command that they know of. Doesn't matter what, doesn't have to be related to any previous post, it just has to be something that they've found useful or can see that others might find useful.

But then I thought, why limit this to just my local LUG list? Sure it's great to test drive the idea, but why not try aiming for a wider audience - so here I am copying my initial get the ball rolling post with some top variants here:


atop - I just had an issue where gnome wasn't logging in, but seemed to be stuck constantly accessing the hard drive. This command saved me by showing me exactly which program was using the hard drive and a quick aptitude remove mlocate later my system was working perfectly again. It monitors CPU, memory, disk and network highlighting any that are particularly stressed and shows the processes responsible. Processes are only displayed if they have done something interesting
since the last update. Kernel patches can be taken to enhance the experience if one is so inclined.

htop - Awesome ncurses graphical top. Looks pretty, coloured, and simply highlighting a process and pressing 'S' will attach strace to it to see what that run away process is actually up to. Tag multiple processes and alter the niceness of them all at once or just kill em' all. 'T' toggles between process tree view and ordinary top view.

powertop - I'm sure lots of people know about this one by now, but for anyone who doesn't it can show you various information about what is chewing up energy in your system and provide some recommendations for conserving power.

iftop - top for network traffic. Shows the traffic going to and fro on every individual transfer and totalled down the bottom in ncurses bar graph style. Amounts are displayed for the last 2, 10 and 40 seconds. Filters can be applied if one is only interested in a subset of the total traffic, and it can naturally do hostname lookups and show port numbers/service names.

ntop - another network top, but this one starts a web server on port 3000 to display it's results with pretty graphs. It has the advantage that it provides much more detail - it breaks packets down by size, protocol, etc. It has many displays to analyse the data in varying and sometimes entertaining ways. Of course, being heavyweight as it is, if all you need to know is that traffic is flowing from A to B, firing this one up may be overkill, though it would easily suit as a very
quick and dirty network monitoring solution.


And a few others that I haven't found so useful myself, but someone else might:

itop - top for interrupts. I can imagine it would be useful for checking if hardware is getting the computers attention when it should be.

jnettop - this is another network top. I prefer iftop since it gives me a graphical display (and it's help page is somewhat more detailed than "I must write something here... :)").

Saturday, 30 August 2008

MythTV scavanges Scrapheap Challenge episodes

My main desktop box has been running this nifty little program called MythTV for over a year now. MythTV is a home-brew Personal Video Recorder for Linux, which essentially means it's kind of like a VCR, but on steroids. It downloads TV guide data from OzTivo and once a week I go through the program guide sorted by genre to see if there is anything on that sounds like it might be worth watching. Then I check it's list of Upcoming Recordings to make sure I don't have any conflicts, resolving them if necessary and go away.
Every now and again when I have some time I bring up it's recorded programs view, pick something to watch, check that it's automatic commercial detection has done it's job making any necessary corrections (which usually takes less than a minute), sit back hit transcode (to permanently remove those commercials as well as reduce the file size) and enjoy the show.
If I'm not at my computer at the time, I can access MythTV remotely using the MythWeb plugin which lets me change my recording schedules remotely and even stream recorded programs to me. It also integrates with the MythMusic plugin which I have found handy on a number of occasions when I've been working on an assignment in a computer lab on campus and wanted some decent music to listen to ;)
Some months ago, I noticed this program in the guide called "Scrapheap Challenge" airing daily on ABC2. The description sounded interesting so I told MythTV to record it on a daily basis. Since then MythTV has not missed a single episode as all of season 9 was aired and then rolling back to the original season 1 from 1998 and showing every episode since (at the time of writing ABC2 is now up to season 7).
Each episode Scrapheap Challenge pits two teams of three members and an expert against each other to build some contraption that has to perform a specific task out of the junk they can find in a scrapheap in only 10 hours (except for some special episodes). The idea for the show came from a scene in Apollo 13 where the astronauts only had a short time to construct a carbon dioxide filter from whatever parts they could find on their space capsule. The show is hosted by Robert Llewellyn (who it took me a long time to realise played Kryten in Red Dwarf all those years ago) and Lisa Rogers (prior to season 5 it was Cathy Rogers) who offer entertaining (and sometimes ridiculous, especially in the episode introductions) commentary throughout the show.
Anyway, the reason for this post is mostly a plug for the show as I have been thoroughly enjoying it and highly recommend it to any aspiring engineer or indeed anyone with a technical mindset.

Sunday, 17 August 2008

It's amazing what technology can do these days

So I was at a friends stopping the night, but unable to sleep I eventually decided to call a cab to take me home. Small dilemma: I didn't want to disturb everyone who was asleep in the house any more than I had to. But not to worry, Nokia came to my rescue!
First, I didn't know any more detail than the suburb I was in so I started Nokia Maps on my N95 (my "Mobile Modem", but that's for another post) which fairly quickly acquired a GPS fix even though I was indoors and gave me the street and even number I was at.
The web browser on the N95 is somewhat limited and although I could browse to the Canberra Cabs website with it and get their phone number, I could not use the online booking facility. Calling them would clearly disturb those sleeping around me (and I'm not even sure it would be possible given the setup I've got), so I pulled out my Nokia N800 Internet Tablet. Using my N95 as a bluetooth modem I pointed the Mozilla based browser to the Canberra Cabs website, filled in my details and within just a few minutes was in a cab spending the rest of my money for the night getting home.

Now to sleep.

Monday, 11 August 2008

BSoD Advertising at LCA08

My good friend Chris just sent me this photo taken at linux.conf.au earlier this year. I'm on the left, Jason's on the right, and well, you can see what's in the middle ;)

Monday, 14 July 2008

N800 as Remote Speaker and Overdue LCA Overview

Well it has been a long time, I really don't have any excuse. So much for blogging about LCA, ey? I actually have a half written blog post about it still sitting on my Zaurus, but I never got around to finishing it. My Internet Tablet functioned beautifully (except for a bug regarding the bluetooth keyboard requiring me to reboot it every now and again to get the on screen keyboard to come back) and was all I really needed there. The only use the Zaurus got was a little bit of showoff, typing that unpublished blog post when I didn't want to pull out that bulky dell keyboard and listen to music on the train when I didn't want to exhaust the N800's batteries. The only time a full sized laptop would have been particularly handy was during some of the tutorials where it was impossible to keep up on the tablet.
It was awesome being able to meet so many people including some of Nokia's employees and seeing the N810 in the flesh, as well as Linus himself. If you must know I saw him four times - after a kernel talk I saw him sneak into an unmarked room, when he walked in on the kernel dev panel talking about kernel debuggers ("Linus! We were just talking about... ice cream.... Would you like some ice cream Linus?"), when he ducked up the stairs to grab a snack and during the meal on the last day.
By now there isn't much point in posting info about any of the talks since it has well and truly been blogged to death. If you haven't already go watch the vids - In particular I recommend Tux' Angels: Incident Response Revealed (about IT forensics using open source tools) and Viktor Olier's talk on the RepRap (I think it was titled The Replicators are Coming or something).

Nightwish was *awesome*. I did have to miss out on the Penguin Dinner that night to make it, but ohh was it worth it. Unfortunately I missed out on a shirt since they had stopped selling them before interval when I was planning on getting one. No matter - I have since ordered some Nightwish and Sonata Arctica merchandise from overseas. Google paying for the bar tab at the Students party followed by gelato was pretty sweet - I think that brings the total meals of mine that Google has paid for to 5 :-)


But anyway, moving on to today's post: I now have found yet another use for my Internet Tablet!

When it's late at night and I'm at college and want to watch a video or listen to music without disturbing my neighbors I use a 5 meter headphone extension cable to reach my bed. I've been meaning to look into a bluetooth headset but haven't got around to it yet. Anyway, I've come back home for a few weeks while uni is on break and it's late at night and I want to watch a video without disturbing my mother. Problem is I forgot to pack my headphone extension cable and it's too cold and uncomfortable to sit next to the computer watching it.
So I started wondering if there was any way that I could use my Internet Tablet as remote speakers across the room - then I could plug my earphones into them and effectively have (almost) wireless earphones. Turns out that it is possible and not too difficult at all.
The two are connected via wireless using my Linksys WRT54GL - which wouldn't be necessary if I could get my laptop to use ad-hoc properly or if either device supported functioning as master. Another option that I haven't looked into yet is using bluetooth PAN, or possibly hacking the tablet to look like a bluetooth A2DP headset, but that's for another time and probably another person to hack into existence.
On my tablet I opened a terminal and ran:
esd -public -tcp -nobeeps
I then opened a second terminal window and SSHd into my laptop. There I ran:
export DISPLAY=:0
export ESPEAKER=192.168.1.101 (My Internet Tablet's IP)
mplayer -ao esd -delay -0.3 video.avi

I had to use the -delay -0.3 parameter as the audio was slightly delayed due to the overhead of sending it over the network and I found that number gave pretty good lip sync - you would probably have to fiddle around with it to find the optimal setting for your situation. This technique should work for any application that can use esd to output sound.