Planet Larry

Computers are a love/hate thing for me. I love all things digital, but I desperately need to get away from it sometimes too. So I had a nice vacation away from my computer last week. I couldn't keep myself from reading some mailing lists and hitting Slashdot once a day, but I didn't write a single line of code and didn't give my websites or work projects or anything much thought.

But now my vacation is over, and it's so easy to fall back into old habits, endlessly looking at webcomics and reading articles about Common Lisp unit testing suites and cringing at the latest drama amongst Gentoo devs and minding my message board like a crusty old beat cop making his rounds. It's the life I've chosen, and I do like it, but I do like getting away sometimes too.

I fulfilled one of my dreams last week when I finally caved and ordered a solid glass mousepad. They're pretty cheap on newegg.com, depending on the color you want. I happened to want green, and it happened to be the cheapest, so all is well. It looks very nice, and it's big and hopefully the surface won't degrade over time; I tend to eat through mousepads via a slow yet inexorable process of erosion.

Unfortunately my laser mouse doesn't work on it. However, I have learned that if I upgrade my mouse's firmware, it will magically be able to work on a solid glass mousepad. Who would've thought my mouse had updateable firmware, let alone that updating the firmware would allow it to work on new surfaces? Not I.

The bad thing is that I need freaking Windows XP to upgrade the firmware on my mouse. I don't have any computer that has XP on it and I'm afraid to try anything in a virtual machine that involves something as dangerous as fiddling with the innards of connected peripherals. So I tried to install XP on my laptop, desperate times calling for desperate measures. But of course the install failed because my XP install CD is so old (pre-SP1, received free from my college 7 years ago) that it didn't recognize most of my hardware. In fact, the XP install CD blue-screened, which set a new record for how low Windows could sink in my opinion.

So I tried slipstreaming SP2 into my install CD. But it failed because, get this, the filenames of some drivers on the CD, namely usbehci.sys, ended up in lower case rather than uppercase and the CD's install program couldn't locate them. I kid you not. Since when is anything in Windows case-sensitive? Is it running Linux? I had to burn another CD after renaming all the files into uppercase. Then the CD worked, but it couldn't find my hard drive, probably due to missing SATA drivers. At that point I gave up, and plan to take my mouse to work tomorrow to upgrade the firmware on a work machine that has XP on it.

And so the score up to this point in my life is Windows: 947, Brian: 0. Windows remains undefeated.

Thanks go out to Logitech for not letting me use Vista (or, say, LINUX) to upgrade my mouse's firmware, and of course to Microsoft, for yet another gloriously broken and frustrating computing experience.

Google’s SketchUp is an incredibly easy-to-use architectural and 3D-modeling tool. The Computer Science Club here at Mansfield has been talking about modeling the university for a 3D perspective in Google Earth, similar to what Google Earth provides for New York City and other metropolitan areas. Recently, I have been experimenting with SketchUp. A free version is available from the website and a professional version can be purchased for about $500. People enrolled in Universities can obtain the professional licenses for $50 a year. The $50 counts towards purchasing a full license, so after 10 years, it’s yours forever. Or, you can pay for the educational license for 3 years and pay the rest of the money up front.

At a first glance, SketchUp seems to be too simple to be worth anything. Upon further investigation, the simplicity seems to come from Google’s innovational perspective. Google has outdone themselves with SketchUp. It is amazingly easy to pick up and create simple objects. More complicated objects can be created with some practice. I have spent about an hour and a half working with the program. First, I watched the beginning tutorials. Then, I went straight into making objects and refining them. It’s amazingly simple. Just make a shape, pull it up to give it depth, draw other objects on it, and manipulate them. It’s amazingly simple.

In about twenty minutes, I was able to make the desk that I use here at school. Keep in mind that it isn’t 100% perfect, nor is it 100% to scale. I have never seen a program this simple. With an hour and a half of experience, I was easily able to make this (click to enlarge):

Here is the front of it:

From an angle:

From the side (notice the arches in the drawers):

And here it is next to a person:

I consider OLPC to be one of the most exciting initiatives of the last few years. When the idea was first circulated it was such an exciting call to arms to do something about the lack of education in poor regions of the world. And the project has produced what appears to be a pretty incredible product, the research of which is now recycled back into the general hardware industry, so it has brought advances that wouldn’t otherwise have happened (now).

I recall pondering the real purpose of the project, asking what is going to be achieved with these laptops. The OLPC project had a very good answer to this. They said the laptops will promote learning in areas where school books are a luxury. Furthermore, the laptop itself is completely tweakable, you press a special key and the source code of the current program pops up. This will promote learning through tweaking and experimentation, so that eventually an industry can be built on these foundations, in regions where little industry exists today and where perhaps the potential for one (in terms of natural resources) is bleak. A beautiful dream, one that could change the world in big ways.

Now Negroponte has changed his tune. Visionary that he is, he failed to convince the clients of the value of free software. So now he’s humming “forget open source, it’s all about the kids!” while preparing to run Windows on the laptop. There is a new smoke screen being constructed:

Negroponte says that the organization is working to ensure that Sugar can run smoothly on Windows.

Riiiight, running Sugar on Windows. Tell me, what exactly is the value of running Windows with an all free software stack? It’s completely useless, that’s what. The whole value of Windows is as a platform, not merely as an operating system. People buy Windows to run Windows applications, not for Windows itself. Or are we actually buying that Egyptian officials are eager to purchase Windows licenses in order to run the free software suite?

Congratulations, Negroponte, you’ve just become a licensed Windows vendor. The kids will no doubt have fun clicking on the Start menu and playing Solitaire. There is a great deal to learn from that, just nothing about the operating system or the applications, you know, actual learning.

OLPC in its original form was about empowering the users, with Windows that capability is entirely destroyed. The fact you cannot mix learning with trade secrets should be blindly obvious to anyone. Open souce is important, but it’s especially important when you want people to learn something.

Furthermore, learning doesn’t happen in isolation. It’s accelerated when it happens in a community of ideas and impulses that flow freely. Resigning OLPC president gets it when he says:

“What comes part and parcel with open source is a culture, and it’s the culture that I’m interested in,” he says. “It’s a culture of expression and critique, sharing, collaboration, appropriation.” And this culture can and should spill into classrooms, he says.

Mal eben zum festhalten:

Einfache Schleife
for i in `ls` ; do echo $i ; done

In Dateien suchen und ersetzen
cat filename | sed ’s/suchen/ersetzen/g’ > filename-new

Dateien umbenennen und hinten etwas wegnehmen
mv $i ${i%-new}

For a very long while now, several years actually, I've been a bit annoyed by the behaviour of terminals under X when you doubleclick links. What the UI considers a word is selected. Selection 'starts' at the point that is doubleclicked and 'spreads' in each direction, stopping at a char it considers to be a word delimiter. A space is probably always considered a delimiter. Sometimes a '?' too, and often ',' as well. This has been very annoying for me as I spend quite alot of time in irc (irssi in a screen, its lovely in combination with bitlbee; icq, irc etcetc in the same screen.) and every now and then someone pastes an url which I want to doubleclick and then paste into my browser. Now the selection stops prematurely as it's not uncommon for a hyperlink to contain one of the characters that is in the word delimiter list. I've thought of this as a limitation to the system I use and, although annoyed, never gave it much thought. Maybe I've been to susceptible to propagandaists telling me gnu/linux is user-unfriendly.

A couple of weeks ago I started to think a little about it and the solution is rather simple. I use Eterm more or less exclusively.

In ~/.Eterm/user.cfg I've put this:

<eterm-0.9>
begin misc
cut_chars "\t\\\`\\\"\'() *,;<>[]{|}"
end misc

This might not be perfect but it's certainly served it's purpose this far as I have been able to doubleclick on the links and then middleclick in the address field (or open a new tab) in the browser. My OS is even better =)

Graduate School Information Request

---

Your request for information has been processed.

Thank you for your interest in the University of Massachusetts Amherst.

Applications directed to addresses within the United States are mailed twice weekly with first class postage; allow 10 days for delivery. Applications directed to non-U.S. addresses are mailed weekly and sent air mail; allow at least 3 weeks for receipt. There is no charge for any of these materials.

The following materials will be sent to you here:

Department of Computer Science
KICT, IIUM
Gombak, 53100 MYS

• CMPSC program materials

Sometimes you may want to check that an email address is not syntactically invalid, i.e. it looks like a recognisable email address. I use this approach in my zetact contact form processor.

Of course, it does not mean the address actually leads anywhere, but at least you know are dealing with an email address that could exist.

This is the code I have been using, albeit I have changed it from a class method to a simple function to make this post simpler.

"""Email check using regex."""

def invalidreg(emailkey):
    """Email validation, checks for syntactically invalid email             
    courtesy of Mark Nenadov.                                               
    See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65215"""
    import re
    emailregex = "^.+\\@(\\[?)[a-zA-Z0-9\\-\\.]+\\.([a-zA-Z]{2,3}|[0-9]{1,3\
})(\\]?)$"
    if len(emailkey) > 7:
        if re.match(emailregex, emailkey) != None:
            return False
        return True
    else:
        return True

I decided it would be more Pythonic to try to do this using the built-in string methods, rather than importing the re module and using a monster regular expression. Here was my first attempt.

"""Email checks using string methods - simple version."""
def invalidemail(emailaddress):
    """Checks for a syntactically invalid email address."""
    try:
        emailitems = emailaddress.rsplit('@', 1)
        emailitems.extend(emailitems[1].rsplit('.', 1))
    except IndexError:
        return True

    if [x for x in emailitems if not x.replace(".","").isalnum()] \
            and emailaddress >= 7:
        return True
    else:
        return False

After a bit of testing and playing with this, a friend pointed me towards the relevant RFC on restrictions of email addresses. While the standard allows the use of many different special characters, in practice email addresses have to be much stricter if you actually want people in the real world to be able to send email to you.

For example, if we allow the email address []@commandline.org.uk, will whatever receives the output of this function be able to use it? As pointed out by Jan Goyvaerts, most software won't actually be able to handle obscure special characters.

We also don't want to water down the syntax check and allow junk for the sake of theoretical but non-existent addresses.

My compromise is to allow these special symbols -_.%+. in the local-part of the email address, and -_. in the domain name. I also do sanity checking on the top-level domain, it needs to be either a generic name or two characters long (country codes are all two letters).

So below is my current version, I added lots of comments and white space to make it easy to read.

"""Ditch nonsense email addresses."""

GENERIC_DOMAINS = "aero", "asia", "biz", "cat", "com", "coop", \
    "edu", "gov", "info", "int", "jobs", "mil", "mobi", "museum", \
    "name", "net", "org", "pro", "tel", "travel"

def invalid(emailaddress, domains = GENERIC_DOMAINS):
    """Checks for a syntactically invalid email address."""

    # Email address must be 7 characters in total. 
    if len(emailaddress) < 7:
        return True # Address too short.

    # Split up email address into parts.
    try:
        localpart, domainname = emailaddress.rsplit('@', 1)
        host, toplevel = domainname.rsplit('.', 1)
    except ValueError:
        return True # Address does not have enough parts. 
    
    # Check for Country code or Generic Domain.
    if len(toplevel) != 2 and toplevel not in domains:
        return True # Not a domain name.
    
    for i in '-_.%+.':
        localpart = localpart.replace(i, "")
    for i in '-_.':
        host = host.replace(i, "")
    
    if localpart.isalnum() and host.isalnum():
        return False # Email address is fine.
    else:
        return True # Email address has funny characters.
    
# Start the ball rolling.
if __name__ == "__main__":
    print invalid("warrior@example.com")

Discuss this post - Leave a comment

[update] Question solved, see bottom of post.

Since Python 2.5 the language got a new built-in method ‘all’ (and it’s nephew ‘any’). I wanted to play around with this a little, combined with generators, so I created a little testcase to test performance.

Here’s the test-case: take a list L of X random numbers in a given range [A, B], and check whether

• all elements in L are >= A
• all elements in L are >= (A + Z) where Z is a number in [0, (B - A)]

The first test should always result True, the second test could result to False.

Here’s the output of a test-run:

In [1]: import random, sys

In [2]: a = [random.randint(100, sys.maxint) for i in xrange(2000000)]

In [3]: len(a)
Out[3]: 2000000

In [4]: #Check whether all elements are >= 100 

In [5]: %timeit all(i >= 100 for i in a)
10 loops, best of 3: 515 ms per loop

In [6]: %timeit any(i < 100 for i in a)
10 loops, best of 3: 454 ms per loop

In [7]: def f(l):
   ...:     for i in l:
   ...:         if i < 100:
   ...:             return False
   ...:     return True
   ...: 

In [8]: %timeit f(a)
10 loops, best of 3: 292 ms per loop

In [9]: #Same thing for 100000, since now the list shouldn't be completely iterated

In [10]: %timeit all(i >= 100000 for i in a)
100 loops, best of 3: 4.73 ms per loop

In [11]: %timeit any(i < 100000 for i in a)
100 loops, best of 3: 4.29 ms per loop

In [12]: def g(l):
   ....:     for i in l:
   ....:         if i < 100000:
   ....:             return False
   ....:     return True
   ....: 

In [13]: %timeit g(a)
100 loops, best of 3: 2.82 ms per loop

In [14]: #For reference

In [15]: %timeit False in (i >= 100 for i in a)
10 loops, best of 3: 531 ms per loop

In [16]: %timeit False in (i >= 100000 for i in a)
100 loops, best of 3: 5.03 ms per loop

It’s as if ‘all’, ‘any’ or ‘in’ don’t break/return when a first occurence of False (or True, obviously) is found. Is this the desired behaviour, and if it is, why? The calculation time difference between using all/any/in or a custom-made function (which is, unlike all etc, not written in C) which breaks whenever it can, is pretty astonishing.

[update] Question solved. It’s pretty normal the function-based approach performs better, since it combines what ‘all’ and the generator provided to ‘all’ do, taking away the generator function-call overhead. Damn

If you’ve been dealing with files for a while you will have noticed that there is a slight semantic gap between how humans see files and how computers do. If you’ve ever seen a file list like this you know what I mean:

Lecture10.pdf
Lecture11.pdf
Lecture12.pdf
Lecture1.pdf
Lecture2.pdf
…

Numbering these files was done in good faith, and a user understands what it means, but the computer doesn’t get it. Sorting in dictionary order produces the wrong order as far as the user is concerned. The reason is that the digits in these filenames are not treated and compared as integers, merely as strings. (Actually, . comes before 0 in ASCII, what’s going on here?)

While we’re not expecting our computers to wisen up about this anytime soon, there is the obvious fix:

Lecture01.pdf
Lecture02.pdf
…
Lecture10.pdf
Lecture11.pdf
Lecture12.pdf

You’ve probably done this by hand once or twice, while cursing.

On the upshot, this is very easy to fix with a few lines of code:

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# revision 1 - support multiple digit runs in filenames
 
import os, string, glob, re, sys
 
def renseq():
    if (len(sys.argv) != 2):
        print "Usage:\\t" + sys.argv[0] + " <num_digits>"
    else:
        ren_seq_files(sys.argv[1])
 
 
def ren_seq_files(num_digits):
    files = glob.glob("*")
    for filename in files:
        m = re.search("(.*)(\\..*)", filename)
        ext = ""
        if m: (filename, ext) = m.groups()
 
        digit_runs = re.finditer("([0-9]+)", filename)
        spans = [m.span() for m in digit_runs if digit_runs]
        if spans:
            spans.reverse()
            arr = list(filename)
            for (s, e) in spans:
                arr[s:e] = string.zfill(str( int(filename[s:e]) ), int(num_digits))
            os.rename(filename+ext, "".join(arr)+ext)

 
 
if __name__ == "__main__":
    renseq()

Download this code: renseq.py

This works on all the files in the current directory. Pass an integer to renseq.py and it will change all the numbers in a filename (if there are any) to the same numbers, padded with zeros if they have fewer digits than the amount you want. So on the example

renseq.py 2

will turn the first list into the second list.

If say, there are filenames with numbers of three digits and you pass 2 to renseq.py, the numbers will be preserved (so it’s not a destructive rename), you’ll just revert to your incorrect ordering as it was in the beginning.

renseq.py will rewrite all the numbers in a filename, but not the extension. So mp3 won’t become mp03.

I had to fix a problem at my dad's company...
"The network was broken."

It was a NetBEUI network connecting some windows stations - it has been running for years - and now suddenly the nodes couldn't find eachother.
One of the boxes (windows 2000 iirc) had 2 network cards, one for the network, the other not used for anything (not even connected). Disabling the latter - not even touching the former - fixed half of the network.

read more

Someone in #lh today told me about Sender Policy Framework, which sounds like a badly-needed enhancement to the Internet’s email protocols. Basically, the idea is to provide a DNS record that informs MTAs “don’t trust emails claiming to be from this domain unless they’re coming from one of my actual servers".

In DNS, this looks like (in my case):

emptymatter.org. IN TXT "v=spf1 a mx ~all"

Some MTAs support SPF but need to be configured, I believe Gentoo’s postfix is one of them. If I’m going to expect other mail servers to support it, I probably should myself. I’ll have to tackle that another day…

Since Heise announced that Microsoft will release the Windows XP Service Pack 3 on the 29th, I didn’t sleep too well, as I really want to grab it as soon as possible. Sure, I use Linux, but this doesn’t mean that I am not dangling with windows boxes from time to time and I am fed up applying about 100 patches before I can even think of security.

But I was laughing too soon - as just one day after the 29th (today), I have seen a posting on the Washington Post Blog that Microsoft has delayed the start of the service pack again. In a written statement they say:

“In order to make sure customers have the best possible experience we have decided to delay releasing Windows XP SP3 to Windows Update and Microsoft Download Center.”

In other words, there seems to be no release date yet. Well - in the meantime I’m rolling my own update pack using the Heise Offline Update. Thanks a ton, guys.

Nothing exciting here. Got my tax refunds. Might build a home-made NAS with a couple terabytes of disk and put it in the basement.

On the DS, I’ve been playing Rondo of Swords and The World Ends With You. Rondo is a pleasant find, a difficult but still reasonable strategy RPG that makes one think and plan ahead, unlike games such as Revenant Wings which are much more “bring a healer and just mob everyone at the thing they’re strong against!” Also, I have a crush on Atlus by this point. There’s no denying it now. I draw their name with little hearts all around when I’m in meetings.

The World Ends With You is refreshingly original, one of those games that, even with it being Square Enix, is a bit surprising that it made it to the States. Very Japanese, and the game makes few concessions to the English audience. Sure, long gone are the times of gratitutous name changes, but even the j-pop/j-rock soundtrack remains intact, and that is, to my slightly jaded mind, a bit commendable. Now, if only the main character didn’t suffer from two vile Square Enix staples: unimaginable thinness and nearly sickening teenage angst. Neku is supposed to get better with the latter; I hope it is soon.

My games to beat are now Etrian Odyssey and Rondo of Swords, one I must beat before Etrian Odyssey II (guess which one) is released here, and the other before the Final Fantasy IV remake reaches the States. I’m excited. If I have time before those, Final Fantasy III and The World Ends With You are my RPGs to beat. FF3 is a cakewalk thus far, but its ease and its crude mechanics compared to Final Fantasy V make it hard to stay with for long.

I didn’t really intend this to become all about video games. I’ve been working on a Gentoo Wiki page for the HP 2133 which has kind of slowed down as most of the parts I’m interested in are supported as best they can be without new versions of drivers, I think. There’s some other hardware that I need to try out (the webcam, for example), but I don’t really care that much, so it’s low priority. Notebooky stuff works.

I have a Waterfield Designs bag coming soon, which I’m excited about. Don’t think it will be suitable for gaming books, but I still have that backpack which is going on 5+ years. The little trooper.

I’ve been meaning to survey the gaming group and associated friends to see what they’re using for IM these days. I think the answer for some is “nothing", with a couple saying “AIM on occasion” or “I idle on Google Talk", so I’ve not really been motivated to test those waters. I want to get a private Jabber conference room running for the group, since the IRC thing kind of sputtered off and died (I still idle there!), but I know it means getting people to switch to Jabber (or at least Google Talk) and then getting them to use a non-Google Talk client (Pidgin, I bet, but maybe Trillian would work). Sigh. If anyone has interest in switching to one network (I highly suggest a Jabber-like ["XMPP” for the techies]), or trying out conferences, or whatever, email/IM me and we’ll play around.

This really is getting rambly, and people might expect me to write long posts all the time. So I’m wrapping this up by saying that spring is finally here, and that’s why it snowed yesterday.

Use Readline shortcuts

At the bash prompt, you can use the default readline keybindings, these are similar to Emacs ones. Many of these are also available within other programs that use readline, such as the Python interpreter.

Here are some useful ones:

Ctrl-A Beginning of Line
Ctrl-E End of Line

Ctrl-U Kill (cut) everything left of cursor
Ctrl-K Kill (cut) everything right of cursor
Ctrl-W Kill (cut) the single word before the cursor
Ctrl-Y Yank (paste) the text back

Ctrl-L Clear Screen
Ctrl-D Exit
Ctrl-R Reverse interactive-search, (attempt to complete what is currently being typed using the history file)

SSH without Passwords

If you login to a remote machine often and you get bored of typing the password, then you can use public key cryptography instead.

The way it works is that the remote machine has a copy of your local machine's public key, it can then use that to check that your local machine is really your machine, and so let you in.

To start with, on the local machine, see if you already have a key pair:

ls ~/.ssh/id_?sa.pub

If not, then make one:

ssh-keygen -t dsa

Now you need to copy your public key to the remote host. On the local machine run:

scp ~/.ssh/id_?sa.pub remotehost:

Now we login to the remote server:

ssh remotehost

Append the public key to your authorized keys file

cat id_?sa.pub >> ~/.ssh/authorized_keys

Now you can login without passwords. Make sure the security of your machines is well thought out. Use disk encyption if possible.

Create a script directory in home directory

I often talk about random Python or bash scripts. The easy way to use them on Linux is to make a dedicated script directory for these.

mkdir ~/bin

Add it to your shell's path. Edit ~/.bashrc and add:

export PATH=$HOME/bin:$PATH

Now all the scripts that you add to ~/bin are always available. This makes things a lot more flexible and fun as you can try out various scripts by dropping them in ~/bin and then deleting them when you are bored of them.

Discuss this post - Leave a comment

So Ubuntu hardy doesn't ship with the handy git wrapper/tool cogito because git has all the features in it incoperated... somewhere...

But documentation is surprisingly sparse. Anyways, if you want to set up a git repo nowadays using just git, it should go as something like follows:

root@server # cd /git
root@server # mkdir newrepo
root@server # chgrp git newrepo
root@server # chmod g+ws newrepo
root@server # cd newrepo
root@server # git init

And if this is a public repository

root@server # touch git-daemon-export-ok

On the client side.

user@client $ cd project
user@client $ git init
user@client $ git add *
user@client $ git commit -m "Initial code dump"
user@client $ git remote add origin ssh://user@git.server.com/git/reponame
user@client $ git push origin master

and after that regular

user@client $ git push

works just fine.

1. Logged into a lab computer via XDMCP: I could've used VNC, but that would've required someone to be logged-in on the lab comp.


2. Using my laptop's PulseAudio as the lab computer's default PulseAudio sink: This makes the lab computer's PulseAudio send all sound to my laptop's PulseAudio by default.


3. Connected to my laptop's DAAP share from the lab computer's Rhythmbox: The music on my laptop becomes accessible from the lab computer's Rhythmbox.

This has probably happened to you. You come to a web page that has links to a bunch of pictures, or videos, or documents that you want to download. Not one or two, but all. How do you go about it? Personally, I use wget for anything that will take a while to download. It’s wonderful, accepts http, https, ftp etc, has options to resume and retry, it never fails. I could just use Firefox, and if it’s small files then I do just that, and click all the links in one fell swoop, then let them all download on their own. But if it’s larger files then it’s not practical. You don’t want to download 20 videos of 200mb each in parallel, that’s no good. If Firefox crashes within the next few hours (which it probably will) then you’ll likely end up with not even one file successfully downloaded. And Firefox doesn’t have a resume function (there is a button but it doesn’t do anything ).

So there is a fallback option: copy all the links from Firefox and queue them up for wget: right click in document, Copy Link Location, right click in terminal window. This is painful and I last about 4-5 links before I get sick of it, download the web page and start parsing it instead. That always works, but I have to rig up a new chain of grep, sed, tr and xargs wget (or a for loop) for every page, I can never reuse that and so the effort doesn’t go a long way.

There is another option. I could use a Firefox extension for this, there are some of them for this purpose. But that too is fraught with pain. Some of them don’t work, some only work for some types of files, some still require some amount of manual effort to pick the right urls and so on, some of them don’t support resuming a download after Firefox crashes. Not to mention that every new extension slows down Firefox and adds another upgrade cycle you have to worry about. Want to run Firefox 3? Oh sorry, your download extension isn’t compatible. wget, in contrast, never stops working. Most limiting of all, these extensions aren’t Unix-y. They assume they know what you want, and they take you from start to end. There’s no way you can plug in grep somewhere in the chain to filter out things you don’t want, for example.

So the problem is eventually reduced to: how can I still use wget? Well, browsers being as lenient as they are, it’s difficult to guarantee that you can parse every page, but you can at least try. spiderfetch, whose name describes its function: spider a page for links and then fetch them, attacks the common scenario. You find a page that links to a bunch of media files. So you feed the url to spiderfetch. It will download the page and find all the links (as best it can). It will then download the files one by one. Internally, it uses wget, so you still get the desired functionality and the familiar output.

If the urls on the page require additional post-processing, say they are .asx files you have to download one by one, grab the mms:// url inside, and mplayer -dumpstream, you at least get the first half of the chain. (Unlikely scenario? If you wanted to download these freely available lectures on compilers from the University of Washington, you have little choice. You could even chain spiderfetch to do both: first spider the index page, download all the .asx files, then spider each .asx file for the mms:// url, print it to the screen and let mplayer take it from there. No more grep or sed. )

Features

• Spiders the page for anything that looks like a url.
• Ability to filter urls for a regular expression (keep in mind this is still Ruby’s regex, so .* to match any character, not * as in file globbing, (true|false) for choice and so on.)
• Downloads all the urls serially, or just outputs to screen (with --dump) if you want to filter/sort/etc.
• Can use an existing index file (with --useindex), but then if there are relative links among the urls, they will need post-processing, because the path of the index page on the server is not known after it has been stored locally.
• Uses wget internally and relays its output as well. Supports http, https and ftp urls.
• Semantics consistent with for url in urls; do wget $url… does not re-download completed files, resumes downloads, retries interrupted transfers.

Limitations

• Not guaranteed to find every last url, although the matching is pretty lenient. If you can’t match a certain url you’re still stuck with grep and sed.
• If you have to authenticate yourself somehow in the browser to be able to download your media files, spiderfetch won’t be able to download them (as with wget in general). However, all is not lost. If the urls are ftp or the web server uses simple authentication, you can still post-process them to: ftp://username:password@the.rest.of.the.url, same for http.

Download spiderfetch:

• spiderfetch.rb

Recipes

To make the use a bit clearer, let’s see some concrete examples.

Recipe: Download the 2008 lectures from Fosdem:

spiderfetch.rb http://www.fosdem.org/2008/media/video 2008.*ogg

Here we use the pattern 2008.*ogg. If you first run spiderfetch with --dump, you’ll see that all the urls for the lectures in 2008 contain the string 2008. Further, all the video files have the extension ogg. And whatever characters come in between those two things, we don’t care.

Recipe: Download .asx => mms videos

Like it or not, sometimes you have to deal with ugly proprietary protocols. Video files exposed as .asx files are typically pointers to urls of the mms:// protocol. Microsoft calls them metafiles. This snippet illustrates how you can download them. First you spider for all the .asx urls, using the pattern \.asx$, which means “match on strings containing .asx as the last characters of the string”. Then we spider each of those urls for actual urls to video files, which begin with mms. And for each one we use mplayer -dumpstream to actually download the video.

#!/bin/bash
 
mypath=$(cd $(dirname $0); pwd)
webpage="$1"
 
for url in $($mypath/spiderfetch.rb $webpage "\\.asx$" --dump); do
	video=$($mypath/spiderfetch.rb $url "^mms" --dump)
	mplayer -dumpstream $video -dumpfile $(basename $video)
done
 

Download this code: asx_spiderfetch.sh

Living Code

David Parker famously said that texts are living, once they leave the pen of the author then they have a life of their own, you never know where the text will end up or how it will be modified. For Python code that is even more true.

The beauty of Python is that you can write code fast, share code and modify code. For this to work, your code needs to be readable. Writing code is easy, reading other people's code is much harder, or even reading your own code after a few months or years has past.

Therefore the aim is to make code as readable as possible, even if it causes a little more work when you write it. The way to make your Python code most readable is to keep to the Style Guide for Python Code, also known as PEP8.

Pylint for the Win

It is far easier to keep your code valid to PEP8 as you go along, than to try to move a large codebase to PEP8 at the end. I recommend the use of a tool called pylint.

Pylint is available from all Linux distributions' package managers (e.g. apt-get install pylint or emerge pylint). Here are some instructions for Windows.

If you have ever made a webpage you probably know about HTML-tidy or the online W3C Validator tool. These tell you everything wrong with your HTML.

Pylint is similar, it goes through and tells you both syntax errors and also how your code differs from the PEP8 standard.

There are some corner cases in which you will need to give pylint the finger, but doing it consciously for good reason is better than because you are sloppy.

PEP8 is better than your crappy style

People often don't use PEP8. This is for a variety of (bad) reasons.

Firstly, sometimes people are tourists from another programming language, they do not know any better so they write their Python code like it was Java or C code.

Secondly, Sometimes people think their (cl)own style is better than PEP8 in some technical way. Well that does not matter. I might have a better way to design a plug socket, but if I implemented my better plug socket, I would not be able to buy any electrical devices.

There can only be one standard, and PEP8 is that standard. If you want to change that standard then bribe, sleep with or kill Guido Van Rossum.

Not following the standard makes your code less readable to others, this prevents the quick reuse that Python is designed for (see above).

If you are a free-software/open-source project, then you particularly should be ashamed if you write hard to read code, because allowing other people to read, understand and modify your code is the whole point.

Lastly, some people don't use PEP8 because the document is too circular and verbose for them to remember. I feel your pain, below are the main points in 12 easy rules.

The 12 commandments

Guido, who brought you out of the land of Visual Basic, out of the land of slavery, spake all these words to thee:

• Module names should be in all lowercase - hello.py.
• Class names should be in CamelCase.
• Methods and functions should be in lower_with_underscores
• Implementation-specific 'private' methods _single_underscore_prefix
• Especially private non-subclassable methods __double_underscore_prefix
• Top level constants (i.e. those that are not in a function or class) should be in BLOCKCAPITALS. Overuse of these constants may make your code less reusable.
• If a variable inside a function or method is so temporary and disposable that you cannot give it a name, then use i for the first one, j for the second and k for third.
• Indentation is four spaces per level. No tabs. If you break this rule then you must be stoned in the village square.
• Lines are never more that 80 characters wide. Tip, break lines with a backward slash \. You do not need to do this if there are parentheses, brackets or braces. Don't add extra parentheses just to break lines, use \ instead.
• Spaces after commas, (green, eggs, and, ham)
• Spaces around operators i = i + 1
• Write docstrings for all public modules, functions, classes, and methods. Python is an international community, so use English for docstrings, object names and comments. If you want to provide local translations then use a proper localisation library.

Discuss this post - Leave a comment

The USA’s 2008 election is the first governmental election of any kind that I will have an opportunity to vote in. This is a very important election for everyone. We face major decisions about the war in Iraq and on many major issues. A lot is at stake in this year’s general election. I have followed this election more than any other. I have been involved in a lot of conversations with people who come from very different cultures and backgrounds. Many of these people support my viewpoints, many of them disagree, and many of them could not care less about politics.

I have been appalled at how little it takes for a certain candidate to obtain an undecided and unopinionated person’s vote. Those of us who are strong conservatives or liberals will not be swayed to vote the opposite way for anything in the world (although this is debatable). However, undecided people are swayed way too easily. I have heard at least ten people tell me that they will vote for Obama because he is “young” and relates to “the younger generation” more than any of the “old” people such as Clinton and McCain. I have had one person tell me that, although they are a liberal, they would not vote for McCain if they were conservative because he is “old” and that he will probably “die in office.”

People disregard the candidate’s actual views and opinions and focus in on the unimportant things. They decide to vote for candidates because they “seem like nice people.” I am not sure if it is the college students’ way or if the entire population is this way, but it’s atrocious. If I wanted to, I could run for President, take all of the “correct” views on all of the issues; that is, the ones that would get me elected. I could completely disagree with all of these views, but I could stand up there, smile, act friendly, cry when I hear a sad story, etc. I could be a complete phony and get some of these idiots to follow me just because I am a down-to-earth and likable character.

What has our country come to? Elect someone because they are honest and they agree with your issues. Elect them because they will lead our country in the right (or should I say correct?) direction. But do NOT elect them because they are nice people and they seem friendly. This is ridiculous. Does anyone want someone in office who is nice but leads our country to demise? I don’t think so.

Amendment: It seems that my post has sparked a large debate over at the Gentoo Forums. Check it out here. It looks like most people agree with my basic argument that people are ignorant. However, many people seem to have some insane (read: extreme socialistic / communistic) ideas on the Gentoo Forums.

• Vim: On-the-fly Code Checker for Vim
  

  
  I don't need to say anything about this -- it's simply wonderful.
  

  
  


• Pidgin: Master password support for pidgin
  

  
  Finally, I won't have to type in three passwords when I start Pidgin. gnome-keyring integration ftw!
  

  
  


• GNOME: Reducing memory fragmentation in GNOME
  

  
  Reduced memory is always more than welcome, and it's not an easy task at all :)
  

  
  


• GNOME: SuperRandom: a predictive listening plugin for Rhythmbox
  

  
  This should be interesting. I currently have to manually form lists of songs I like to listen to. Time will tell how well this project fruits.
  

  
  


• GNOME: Make Evolution use Seahorse widgets for encryption and key selection
  

  
  No more UI blocking, and no more f-ugly password prompts! Fun, fun, fun =)
  

  
  


• GNOME: Integrated Media Management for GNOME
  

  
  This has always been a major missing feature in GNOME. This gets implemented, and I owe the student+mentor(s) a beer each *g*
  

  
  


• GStreamer: Media service to transcode video and audio in different formats
  

  
  If this integrates well with the above GNOME project, things will take a turn for the better w.r.t media in GNOME.
  

  
  


• Moinmoin: Extending moin's groups to LDAP and other external sources.
  

  
  We use MoinMoin for our internal wiki, and authentication is done against LDAP. So needless to say, this project will make management of roles using LDAP groups quite straightforward.
  

  
  


• BZFlag: BZFWeb, a web-based BZFS admin interface.
  

  
  Just today we were talking about implementing a basic web interface for BZFlag in Django over the summer, and this thing comes up :)
  Although our idea was to let people select server options using the webinterface, and then connect to the server, this project would be a good thing to keep an eye on.
  

  
  


• FFmpeg: Generic frame-level multithreading support
  

  
  This will rock for everyone with more than one core. This doesn't include me, but includes the Navya Server ;)
  

  
  

Did you hear the exciting news? JRuby 1.1 is out! For real, you can call your grandma with the great news. Wow, that was quick.

Okay, so the big new thing in JRuby is a bytecode compiler. As you may know, up to 1.0 it was just a Ruby interpreter in Java. Now you can actually compile Ruby modules to Java classes and no one will know the difference, very devious. Sounds like Robin Hood in a way, doesn’t it?

The JRuby guys are claiming that this makes JRuby on par with “regular Ruby” on performance, if not better. Hmm. Just to be on the safe side, what size shoes do you wear? Oh ouch, those are going to be tricky to fit in your mouth. And Freud will say you’re stuck in the oral stage. Too much? Okay.

So here is my completely unvetted, dirty, real world test. No laboratory conditions here, you’re in the ghetto. First we need something *to* test. I don’t have a great deal of Ruby code at my disposal, but this should do the trick. How does scanning the raw filesystem for urls sound? The old harvest script actually does a half decent job of turning up a bunch of findings.

Now introducing the contenders. First up, his name is JRuby, you know him from occasional mentions on obscure blogs and the programming reddit past the top 500 entries. He promises to free all Java slaves by giving away free Rubies to everyone!

Aaand the incumbent, the famous… Ruby! You know him, your parents know him, every family would adopt him as their own child if they could. He’s the destroyer of kingdoms and the creator of empires, he’s bigger than Moses himself!

Our two drivers will be racing across a hostile territory. Your track is a 25gb ext3 live file system. During this time, I can promise you that only Firefox is likely to be writing new urls to disk, but I could be lying eheheh. Due to the unpredictable nature of this rally track, regulations allow only one racer at a time, but you will be clocked.

First up is the new kid on the block Jay….Ruby. The Ruby code will not be compiled before execution, we’ll let the just-in-time compiler do its thing.

$ time ( sudo cat /dev/sda5 | bin/jruby harvest.rb –url > /tmp/fsurls.jruby )
real 39m26.547s
user 37m19.072s
sys 1m28.406s

Not too shabby for a first run, but since this a brand new venue, we have no frame of reference yet. Let’s see how Ruby will do here.

$ time ( sudo cat /dev/sda5 | harvest.rb –url > /tmp/fsurls.ruby )
real 78m42.186s
user 62m12.537s
sys 2m18.721s

Well, look at that! The new kid is pretty slick, isn’t he? Sure is giving the old man a run for his money. Let’s see how they answered the questions.

$ lh
-rw-r–r– 1 alex alex 86M 2008-04-21 18:29 fsurls.jruby
-rw-r–r– 1 alex alex 8.6G 2008-04-21 20:58 fsurls.ruby

Yowza! No less than a hundred times more matches with Ruby. What is going on here? Did Jay just race to the finish line, dropping the vast majority of his parcels? Or did father Ruby see double and triple and quadruple, ending up with lots and lots of duplicates? Well, we don’t really *know* how many urls exist in those 25gb of data, but it seems a little bit suspect that there would be in excess of 8gb of them.

One way or the other, it’s pretty clear that the regular expression semantics are not entirely identical. In fact, you might be sweating a little right now if your code uses them heavily.

UPDATE: Squashing duplicates in both files actually produces two files of very similar size (13mb), in which the disparity of unique entries is only a very reasonable 4% (considering the file system was being written to in the process). The question still remains how did Ruby produce 8gb of output.