Planet Larry

Right, so when doing C development on Ubuntu I suddenly noticed something was missing in the pan pages department. Like all the C api.

apt-get install manpages-dev

ah that's better. Someone might want to make it part of the 'build-essentials' package.

The Spring '08 semester has finally drawn to a close. I'm so glad I'll be able to go back to getting reasonable amounts of sleep, and not going to school for several hours a day, then working close to 40 hours a week as well. I'll be much happier being able to split my time between BIOSLEVEL.com and Buffalo Wild Wings.

For BIOSLEVEL, Sapphire recently sent us new AMD 780G chipset motherboard, Radeon HD3450, and an overclocked Radeon HD3870. Between the motherboard and Radeon HD3870, there's a pretty powerful machine just waiting to be powered. Several companies sent us additional parts for the review, which included 2GB of RAM, a quad-core Phenom CPU, and a slew of heatsinks. The reviews of the motherboard and videocard have been posted, and I've got several reviews lined up for the donated components as well. I'm just amazed at the piled of stuff that was shipped to us to power the motherboard.

Simply put, I'm surprised at how well the motherboard's ATI-based integrated graphics performed in Linux. This won't be a motherboard I let out of my sight anytime soon. Since my last final on Wednesday, I've completed an entire two reviews in a single 24-hour period. It's so ridiculously nice to have time away from school again.

Maybe this'll finally be that summer I do something truly productive (and profitable). I've got all the ideas lined up, but do I have the time or motivation to do it? Time will tell.

In Linux when I use SSH I usually pass the host and port and username on the command line and then type the password when prompted. (In those rare cases I don't use certificates to log in without a password.) In Windows, PuTTY makes you pick a host and port and then prompts you for the username AND password.

This leads to unpleasant results. I'm so conditioned to open SSH and type my password at the prompt and hit Enter that I often end up typing my password as my username in PuTTY. Bad.

I've sometimes opened webpages that have some stupid Javascript bullcrap that tries to auto-focus the username field in a login form. But if you're a fast typist (and mouse-ist) like I am, you can focus the field, type your username, and hit tab to get to the password field before the long-loading Javascript bloat has a time to load and run. Which can result in auto-re-focusing the username field, which if it happens at just the right instant, results in my typing the password into it and pounding Enter before I have a chance to notice what's happened. Bad bad bad.

I use a computer far too often to have time too read every prompt, which leads to bad things. Anyone who's used to flying around an interface at light-speed by instinct and repeated learned behavior has experienced this kind of thing I'm sure.

This is horrendously bad because these programs often log the usernames of login attempts in plaintext in logs that lots of potentially evil people have the ability to read. The logs don't usually log the passwords of login attempts, but if you type a password AS a username, oops, you're screwed. Thankfully I'm root on most or all of the machines I ever SSH to, and I can go into /var/log and erase my mistake from the logs before anyone can see. But that doesn't help for web pages I don't know. And I wonder how often this kind of thing happens to other people. I wonder how many people who aren't familiar with computers accidentally send their password as their username to a bunch of websites.

After all the effort we go to to try to secure computer applications, these kinds of stupid human factors can still so easily ruin everything.

The Good

Firefox 3 Beta

Wubi

The Bad

Bootsplash

Drive encryption

The Ugly

Pulseaudio

Summary

A few days ago, I spoke with John Phillips (he is a computer science professor and adviser here at Mansfield) and I found out that I had made three class-selection errors in my Fall 2008 schedule. Luckily, he was able to set me on the right track. My new schedule is linked below. Instead of Accounting II (sorry to Dave and Vinnie), Data Structures (sorry to Dan McKee), and Honors Research (sorry Sara, Andrew, and others), I will be taking Computer Organization, Finite Math, and Principles of Microeconomics.

Revised Fall 2008 Schedule (PDF).

I just visited Sourceforge with the intention of declaring my “Urdabrunn” project as dead. But instead, I decided to announce that the project has moved to my own domain, Solumslekt. The Exodus project is the natural heir and successor of Urdabrunn, and it seems natural to host it here.

The problem with the Urdabrunn project was that it was far too ambitious, like so many other dead projects on Sourceforge. It’s of course much easier to concoct grand plans than to do the actual work.

I did write some routines in Python for the Urdabrunn project, but the major work was the construction and migration of the database itself and the parts of the business logic that I wrote in plpgsql. That codebase has survived to the present day. When I decided to “go incremental” with my humble PHP reports, the interface took off in a totally different direction from what I had imagined at the time I wrote the “Into the Blue” article and submitted the Urdabrunn project to Sourceforge.

For my own part, I feel that Exodus gives me what I need from a genealogy software package. For my personal use, it’s stable to the extent that several months may pass between each time I do even small incremental changes to the codebase.

During the last week, however, I have been in a flurry of refactoring and adding translation strings. That’s because I finally have been engaged in a serious discussion about the project. There is a big difference between writing software for your own use, and having another person trying it out and asking intelligent questions.

After the big download, I kick-started the installer and I couldn’t believe my eyes: That service pack wants to have 1475 MB free space while installation - true, that number sounds scary, but it wasn’t better with the second service pack.

Since the machine boots again, I am looking through application changes - in other words, what does not work any more. But luckily it is only the ATI-Catalyst 8.2 having problems: screen rotation does not work any more. I have not spotted anything else acting strange by now…

After one week delay, the third service pack for Microsoft Windows XP arrived. You won’t believe it: It’s even available via automatic update.

But what’s inside? First of all, it’s a huge file being 313 MB which means a rather big download and a longer installation. But what’s under the hood? Can’t tell by now. I’m just installing.

http://world.std.com/~franl/worm.html

A Slashdot article reminded me of one of my favorite technical articles on the Internet entitled “A Tour of the Worm", an in-depth historical and technical look at the Morris worm. The Morris worm, mistakenly unleashed in 1988, was one of the first significant worms to strike the Internet, and it caused enough damage that it arguably has done the most relative damage of any worm since then.

Check it out if you haven’t read it, or even if you have; it’s a fascinating look at the early days of the Internet. http://world.std.com/~franl/worm.html

Here’s the thing. I hate stating the obvious. It really annoys me. On the other hand, obvious things are sometimes things that most need to be repeated. So I wrestle with myself and I finally decide that I should, because there is a shockingly large number of people out there who don’t realize how obvious this is. See if you can learn something from this mock dialog.

Vendor: Good morning, is this Harry, the CTO*, I’m speaking to?
Client: Yes, how may I help you?
Vendor: Hey Harry, this is Steve from Microsoft. I would like to talk to you about Windows Vista.
Client: What’s that?
Vendor: Why, it’s the brand new version of our Windows operating system.
Client: Oh, that.
Vendor: I was wondering if I could interest you in our product.
Client: You know what, I don’t think so, we are a very security sensitive company, and..
Vendor: But that’s precisely the reason I’m calling, I would like to tell you how you can enhance your security with Windows Vista. You see, we’ve built the operating system with security in mind and it’s the state of the art in operating systems.
Client: Hey, that sounds pretty exciting. So how does this work now, you ship us the source code and…
Vendor: No no, we don’t distribute the source code.
Client: You don’t?!?
Vendor: No, you see it’s a trade secret. (my precious etc)
Client: You’re kidding, right?
Vendor: No, really.
Client: So how do we know that it’s actually secure if we can’t see for ourselves? How do we know there isn’t anything malicious in it?
Vendor: Well you’ll just have to trust us.
*Harry hangs up*
Vendor: Hello? Harry?
*CTO - the highest placed person who makes technical decisions in a company.

How did it go? Did you get it? It was kind of a long thing, huh? Ok, stop racking your brains, I’ll give you the answer: no source code, no security.

Here’s how that works. It’s simple economics, so try to keep up. If they give you the source code, then they put their cards on the table. You can see what the code does, and if it’s doing something stupid (security hole) or nasty (like sending your data to back to the vendor), then you’ll be able to check for this. Now you may say “I don’t know how to check”, and that’s okay. But just by giving you the source code the vendor knows that you can see everything the code is doing. And if you find something nasty in there, they know you’ll never trust them again. So it doesn’t really matter if *you* don’t know how to check, because there are others who do, and sooner or later someone will find the nasty code if it’s in there. Thus, if the vendor gives you the source code, then he’ll be a lot more careful about what’s in there, because he’s risking losing your trust and your business forever. That will keep him honest.

Is there then anything surprising about finding out that Microsoft is putting in backdoors in Windows? No, because how would you know it’s there? You don’t have the source code! In case you were wondering, the words “security” and “backdoor” are mutually exclusive.

So what have we learned today? Is there somehow we could summarize all this in just one sentence? There is: If you want security, ask for the source code. If you can’t get the source code, you know that the vendor isn’t taking security seriously.

  * ERROR: net-fs/nfs-utils-1.1.2-r1 failed.
 * Call stack:
 *       misc-functions.sh, line 652:  Called install_qa_check
 *       misc-functions.sh, line 360:  Called die
 * The specific snippet of code:
 *   		[[ ${abort} == "yes" ]] && hasq stricter ${FEATURES} && die "poor code kills airplanes"
 *  The die message:
 *   poor code kills airplanes

I had to undelete someone's files from a FAT partition today. My first thought was to use good ol' Windows to do so, given that Windows is the unholy ground which spawned FAT to begin with. I remember there used to be an UNDELETE command of some sort in some old version of DOS. But this doesn't seem to exist in XP any longer.

There are however lots and lots of third-party "shareware" programs which can do this kind of thing, as Google reveals. There is in fact an overwhelming number of such shareware programs. Most of these programs are total crap and cost around $30. One program required me to burn a CD and reboot my computer from the CD before I could run it. Many of the programs "intelligently" scan a partition looking for chunks of things that look like JPEGS or WMVs. I tried a few "demos" before I gave up, not having an hour to waste finding the one program that would work. Thus bringing the current score to Windows: 948, Brian: 0.

Instead I brought the drive home and plugged it into Gentoo and used this post as a guide. I dd'ed the partition to a file, fscked around with it a bit, mounted it via loopback, and had my files back. Took 10 minutes, and worked as expected. And it didn't cost me $30.

The moral of this story: I need to burn a Knoppix disk to take to work with me.

My only quibble is that I can never ever remember what Gentoo package contains fsck.vfat. Note to self, it's dosfstools. I can never think of the search terms even to locate that package. I had to google it.

My ninth call to Westinghouse today, about my Westinghouse L2410NM 24" LCD monitor which I RMA'ed back in March, revealed that they did in fact shipmy monitor, supposedly to my house, on April 4th or so. A UPS tracking number confirms it. There are are a few things wrong with this.

1. In spite of the fact that I asked for a phone call to be updated on the status of my monitor whenever it was shipped, I received no such phone call.
2. During the four phone calls (or was it five?) I made to Westinghouse in April, AFTER my monitor was supposedly shipped to my house, no one at the company had any record that it shipped. I was told that by multiple representatives over the past four weeks that my monitor was "in processing".
3. I asked for my monitor to be shipped to workplace, not my house. My nice, safe, cozy workplace with human beings who can sign for large expensive packages. Not my empty house in a neighborhood full of drug addicts, in the property theft capital of the west. In addition to telling the phone representative this, I actually taped a 8.5 x 11 inch sheet of paper directly to the monitor itself (as well as the outside of the box) specifying SHIP TO: and my work address. Even such drastic measures were not enough to catch the attention of whatever magical monitor-repair fairies work at Westinghouse, apparently. Perhaps I should've carved that information directly into the monitor screen.
4. I could possibly overlook the above, except that, as you may have surmised, at the present time, I do not, in fact, have my monitor.

After calling up UPS to ask why their driver left a $450 computer monitor, in a shiny bright blue and white box with pictures of a computer monitor all over it, sitting on my front porch while I was at work without getting my signature, I placed call number ten (yes, I've finally hit double digits!) to Westinghouse, and managed to escalate my issue to the Westinghouse corporate office. Supposedly in 7-10 business days they will send me a brand new monitor.

Oh how I wish I had any confidence that I'm ever going to see that monitor.

In the meantime, this guy was on sale at the local store, so I bought one. Time will tell whether LG brand is any better than Westinghouse. This time, I also bought the extended warranty, having learned my lesson that it can, indeed, be worth an extra $60 to save myself some pain and aggravation later. I'm also going to think twice about buying things like this over the internet in the future. There is something to be said about being able to drive 10 minutes down the road to have your property serviced or replaced by real-life human beings, rather than paying to have things shipped around the world for a month.

I’ve made a difficult decision that I hope doesn’t make anyone feel too bad: effective immediately I’ve removed any feeds from Planet Larry / Larry the Universe that were from developers who have retired from the Gentoo project.

I feel like ex-developers carry a lot of weight with their posts and opinions, and I created the planet feeds mainly for users, by users (speaking of which, don’t be shy, and sign up).

That’s all for now.

"Intellectual Property is the oil of the 21st century"

Computers are a love/hate thing for me. I love all things digital, but I desperately need to get away from it sometimes too. So I had a nice vacation away from my computer last week. I couldn't keep myself from reading some mailing lists and hitting Slashdot once a day, but I didn't write a single line of code and didn't give my websites or work projects or anything much thought.

But now my vacation is over, and it's so easy to fall back into old habits, endlessly looking at webcomics and reading articles about Common Lisp unit testing suites and cringing at the latest drama amongst Gentoo devs and minding my message board like a crusty old beat cop making his rounds. It's the life I've chosen, and I do like it, but I do like getting away sometimes too.

I fulfilled one of my dreams last week when I finally caved and ordered a solid glass mousepad. They're pretty cheap on newegg.com, depending on the color you want. I happened to want green, and it happened to be the cheapest, so all is well. It looks very nice, and it's big and hopefully the surface won't degrade over time; I tend to eat through mousepads via a slow yet inexorable process of erosion.

Unfortunately my laser mouse doesn't work on it. However, I have learned that if I upgrade my mouse's firmware, it will magically be able to work on a solid glass mousepad. Who would've thought my mouse had updateable firmware, let alone that updating the firmware would allow it to work on new surfaces? Not I.

The bad thing is that I need freaking Windows XP to upgrade the firmware on my mouse. I don't have any computer that has XP on it and I'm afraid to try anything in a virtual machine that involves something as dangerous as fiddling with the innards of connected peripherals. So I tried to install XP on my laptop, desperate times calling for desperate measures. But of course the install failed because my XP install CD is so old (pre-SP1, received free from my college 7 years ago) that it didn't recognize most of my hardware. In fact, the XP install CD blue-screened, which set a new record for how low Windows could sink in my opinion.

So I tried slipstreaming SP2 into my install CD. But it failed because, get this, the filenames of some drivers on the CD, namely usbehci.sys, ended up in lower case rather than uppercase and the CD's install program couldn't locate them. I kid you not. Since when is anything in Windows case-sensitive? Is it running Linux? I had to burn another CD after renaming all the files into uppercase. Then the CD worked, but it couldn't find my hard drive, probably due to missing SATA drivers. At that point I gave up, and plan to take my mouse to work tomorrow to upgrade the firmware on a work machine that has XP on it.

And so the score up to this point in my life is Windows: 947, Brian: 0. Windows remains undefeated.

Thanks go out to Logitech for not letting me use Vista (or, say, LINUX) to upgrade my mouse's firmware, and of course to Microsoft, for yet another gloriously broken and frustrating computing experience.

Google’s SketchUp is an incredibly easy-to-use architectural and 3D-modeling tool. The Computer Science Club here at Mansfield has been talking about modeling the university for a 3D perspective in Google Earth, similar to what Google Earth provides for New York City and other metropolitan areas. Recently, I have been experimenting with SketchUp. A free version is available from the website and a professional version can be purchased for about $500. People enrolled in Universities can obtain the professional licenses for $50 a year. The $50 counts towards purchasing a full license, so after 10 years, it’s yours forever. Or, you can pay for the educational license for 3 years and pay the rest of the money up front.

At a first glance, SketchUp seems to be too simple to be worth anything. Upon further investigation, the simplicity seems to come from Google’s innovational perspective. Google has outdone themselves with SketchUp. It is amazingly easy to pick up and create simple objects. More complicated objects can be created with some practice. I have spent about an hour and a half working with the program. First, I watched the beginning tutorials. Then, I went straight into making objects and refining them. It’s amazingly simple. Just make a shape, pull it up to give it depth, draw other objects on it, and manipulate them. It’s amazingly simple.

In about twenty minutes, I was able to make the desk that I use here at school. Keep in mind that it isn’t 100% perfect, nor is it 100% to scale. I have never seen a program this simple. With an hour and a half of experience, I was easily able to make this (click to enlarge):

Here is the front of it:

From an angle:

From the side (notice the arches in the drawers):

And here it is next to a person:

I consider OLPC to be one of the most exciting initiatives of the last few years. When the idea was first circulated it was such an exciting call to arms to do something about the lack of education in poor regions of the world. And the project has produced what appears to be a pretty incredible product, the research of which is now recycled back into the general hardware industry, so it has brought advances that wouldn’t otherwise have happened (now).

I recall pondering the real purpose of the project, asking what is going to be achieved with these laptops. The OLPC project had a very good answer to this. They said the laptops will promote learning in areas where school books are a luxury. Furthermore, the laptop itself is completely tweakable, you press a special key and the source code of the current program pops up. This will promote learning through tweaking and experimentation, so that eventually an industry can be built on these foundations, in regions where little industry exists today and where perhaps the potential for one (in terms of natural resources) is bleak. A beautiful dream, one that could change the world in big ways.

Now Negroponte has changed his tune. Visionary that he is, he failed to convince the clients of the value of free software. So now he’s humming “forget open source, it’s all about the kids!” while preparing to run Windows on the laptop. There is a new smoke screen being constructed:

Negroponte says that the organization is working to ensure that Sugar can run smoothly on Windows.

Riiiight, running Sugar on Windows. Tell me, what exactly is the value of running Windows with an all free software stack? It’s completely useless, that’s what. The whole value of Windows is as a platform, not merely as an operating system. People buy Windows to run Windows applications, not for Windows itself. Or are we actually buying that Egyptian officials are eager to purchase Windows licenses in order to run the free software suite?

Congratulations, Negroponte, you’ve just become a licensed Windows vendor. The kids will no doubt have fun clicking on the Start menu and playing Solitaire. There is a great deal to learn from that, just nothing about the operating system or the applications, you know, actual learning.

OLPC in its original form was about empowering the users, with Windows that capability is entirely destroyed. The fact you cannot mix learning with trade secrets should be blindly obvious to anyone. Open souce is important, but it’s especially important when you want people to learn something.

Furthermore, learning doesn’t happen in isolation. It’s accelerated when it happens in a community of ideas and impulses that flow freely. Resigning OLPC president gets it when he says:

“What comes part and parcel with open source is a culture, and it’s the culture that I’m interested in,” he says. “It’s a culture of expression and critique, sharing, collaboration, appropriation.” And this culture can and should spill into classrooms, he says.

Mal eben zum festhalten:

Einfache Schleife
for i in `ls` ; do echo $i ; done

In Dateien suchen und ersetzen
cat filename | sed ’s/suchen/ersetzen/g’ > filename-new

Dateien umbenennen und hinten etwas wegnehmen
mv $i ${i%-new}

For a very long while now, several years actually, I've been a bit annoyed by the behaviour of terminals under X when you doubleclick links. What the UI considers a word is selected. Selection 'starts' at the point that is doubleclicked and 'spreads' in each direction, stopping at a char it considers to be a word delimiter. A space is probably always considered a delimiter. Sometimes a '?' too, and often ',' as well. This has been very annoying for me as I spend quite alot of time in irc (irssi in a screen, its lovely in combination with bitlbee; icq, irc etcetc in the same screen.) and every now and then someone pastes an url which I want to doubleclick and then paste into my browser. Now the selection stops prematurely as it's not uncommon for a hyperlink to contain one of the characters that is in the word delimiter list. I've thought of this as a limitation to the system I use and, although annoyed, never gave it much thought. Maybe I've been to susceptible to propagandaists telling me gnu/linux is user-unfriendly.

A couple of weeks ago I started to think a little about it and the solution is rather simple. I use Eterm more or less exclusively.

In ~/.Eterm/user.cfg I've put this:

<eterm-0.9>
begin misc
cut_chars "\t\\\`\\\"\'() *,;<>[]{|}"
end misc

This might not be perfect but it's certainly served it's purpose this far as I have been able to doubleclick on the links and then middleclick in the address field (or open a new tab) in the browser. My OS is even better =)

Graduate School Information Request

---

Your request for information has been processed.

Thank you for your interest in the University of Massachusetts Amherst.

Applications directed to addresses within the United States are mailed twice weekly with first class postage; allow 10 days for delivery. Applications directed to non-U.S. addresses are mailed weekly and sent air mail; allow at least 3 weeks for receipt. There is no charge for any of these materials.

The following materials will be sent to you here:

Department of Computer Science
KICT, IIUM
Gombak, 53100 MYS

• CMPSC program materials

Sometimes you may want to check that an email address is not syntactically invalid, i.e. it looks like a recognisable email address. I use this approach in my zetact contact form processor.

Of course, it does not mean the address actually leads anywhere, but at least you know are dealing with an email address that could exist.

This is the code I have been using, albeit I have changed it from a class method to a simple function to make this post simpler.

"""Email check using regex."""

def invalidreg(emailkey):
    """Email validation, checks for syntactically invalid email             
    courtesy of Mark Nenadov.                                               
    See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65215"""
    import re
    emailregex = "^.+\\@(\\[?)[a-zA-Z0-9\\-\\.]+\\.([a-zA-Z]{2,3}|[0-9]{1,3\
})(\\]?)$"
    if len(emailkey) > 7:
        if re.match(emailregex, emailkey) != None:
            return False
        return True
    else:
        return True

I decided it would be more Pythonic to try to do this using the built-in string methods, rather than importing the re module and using a monster regular expression. Here was my first attempt.

"""Email checks using string methods - simple version."""
def invalidemail(emailaddress):
    """Checks for a syntactically invalid email address."""
    try:
        emailitems = emailaddress.rsplit('@', 1)
        emailitems.extend(emailitems[1].rsplit('.', 1))
    except IndexError:
        return True

    if [x for x in emailitems if not x.replace(".","").isalnum()] \
            and emailaddress >= 7:
        return True
    else:
        return False

After a bit of testing and playing with this, a friend pointed me towards the relevant RFC on restrictions of email addresses. While the standard allows the use of many different special characters, in practice email addresses have to be much stricter if you actually want people in the real world to be able to send email to you.

For example, if we allow the email address []@commandline.org.uk, will whatever receives the output of this function be able to use it? As pointed out by Jan Goyvaerts, most software won't actually be able to handle obscure special characters.

We also don't want to water down the syntax check and allow junk for the sake of theoretical but non-existent addresses.

My compromise is to allow these special symbols -_.%+. in the local-part of the email address, and -_. in the domain name. I also do sanity checking on the top-level domain, it needs to be either a generic name or two characters long (country codes are all two letters).

So below is my current version, I added lots of comments and white space to make it easy to read.

"""Ditch nonsense email addresses."""

GENERIC_DOMAINS = "aero", "asia", "biz", "cat", "com", "coop", \
    "edu", "gov", "info", "int", "jobs", "mil", "mobi", "museum", \
    "name", "net", "org", "pro", "tel", "travel"

def invalid(emailaddress, domains = GENERIC_DOMAINS):
    """Checks for a syntactically invalid email address."""

    # Email address must be 7 characters in total. 
    if len(emailaddress) < 7:
        return True # Address too short.

    # Split up email address into parts.
    try:
        localpart, domainname = emailaddress.rsplit('@', 1)
        host, toplevel = domainname.rsplit('.', 1)
    except ValueError:
        return True # Address does not have enough parts. 
    
    # Check for Country code or Generic Domain.
    if len(toplevel) != 2 and toplevel not in domains:
        return True # Not a domain name.
    
    for i in '-_.%+.':
        localpart = localpart.replace(i, "")
    for i in '-_.':
        host = host.replace(i, "")
    
    if localpart.isalnum() and host.isalnum():
        return False # Email address is fine.
    else:
        return True # Email address has funny characters.
    
# Start the ball rolling.
if __name__ == "__main__":
    print invalid("warrior@example.com")

Discuss this post - Leave a comment

[update] Question solved, see bottom of post.

Since Python 2.5 the language got a new built-in method ‘all’ (and it’s nephew ‘any’). I wanted to play around with this a little, combined with generators, so I created a little testcase to test performance.

Here’s the test-case: take a list L of X random numbers in a given range [A, B], and check whether

• all elements in L are >= A
• all elements in L are >= (A + Z) where Z is a number in [0, (B - A)]

The first test should always result True, the second test could result to False.

Here’s the output of a test-run:

In [1]: import random, sys

In [2]: a = [random.randint(100, sys.maxint) for i in xrange(2000000)]

In [3]: len(a)
Out[3]: 2000000

In [4]: #Check whether all elements are >= 100 

In [5]: %timeit all(i >= 100 for i in a)
10 loops, best of 3: 515 ms per loop

In [6]: %timeit any(i < 100 for i in a)
10 loops, best of 3: 454 ms per loop

In [7]: def f(l):
   ...:     for i in l:
   ...:         if i < 100:
   ...:             return False
   ...:     return True
   ...: 

In [8]: %timeit f(a)
10 loops, best of 3: 292 ms per loop

In [9]: #Same thing for 100000, since now the list shouldn't be completely iterated

In [10]: %timeit all(i >= 100000 for i in a)
100 loops, best of 3: 4.73 ms per loop

In [11]: %timeit any(i < 100000 for i in a)
100 loops, best of 3: 4.29 ms per loop

In [12]: def g(l):
   ....:     for i in l:
   ....:         if i < 100000:
   ....:             return False
   ....:     return True
   ....: 

In [13]: %timeit g(a)
100 loops, best of 3: 2.82 ms per loop

In [14]: #For reference

In [15]: %timeit False in (i >= 100 for i in a)
10 loops, best of 3: 531 ms per loop

In [16]: %timeit False in (i >= 100000 for i in a)
100 loops, best of 3: 5.03 ms per loop

It’s as if ‘all’, ‘any’ or ‘in’ don’t break/return when a first occurence of False (or True, obviously) is found. Is this the desired behaviour, and if it is, why? The calculation time difference between using all/any/in or a custom-made function (which is, unlike all etc, not written in C) which breaks whenever it can, is pretty astonishing.

[update] Question solved. It’s pretty normal the function-based approach performs better, since it combines what ‘all’ and the generator provided to ‘all’ do, taking away the generator function-call overhead. Damn

If you’ve been dealing with files for a while you will have noticed that there is a slight semantic gap between how humans see files and how computers do. If you’ve ever seen a file list like this you know what I mean:

Lecture10.pdf
Lecture11.pdf
Lecture12.pdf
Lecture1.pdf
Lecture2.pdf
…

Numbering these files was done in good faith, and a user understands what it means, but the computer doesn’t get it. Sorting in dictionary order produces the wrong order as far as the user is concerned. The reason is that the digits in these filenames are not treated and compared as integers, merely as strings. (Actually, . comes before 0 in ASCII, what’s going on here?)

While we’re not expecting our computers to wisen up about this anytime soon, there is the obvious fix:

Lecture01.pdf
Lecture02.pdf
…
Lecture10.pdf
Lecture11.pdf
Lecture12.pdf

You’ve probably done this by hand once or twice, while cursing.

On the upshot, this is very easy to fix with a few lines of code:

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# revision 1 - support multiple digit runs in filenames
 
import os, string, glob, re, sys
 
def renseq():
    if (len(sys.argv) != 2):
        print "Usage:\\t" + sys.argv[0] + " <num_digits>"
    else:
        ren_seq_files(sys.argv[1])
 
 
def ren_seq_files(num_digits):
    files = glob.glob("*")
    for filename in files:
        m = re.search("(.*)(\\..*)", filename)
        ext = ""
        if m: (filename, ext) = m.groups()
 
        digit_runs = re.finditer("([0-9]+)", filename)
        spans = [m.span() for m in digit_runs if digit_runs]
        if spans:
            spans.reverse()
            arr = list(filename)
            for (s, e) in spans:
                arr[s:e] = string.zfill(str( int(filename[s:e]) ), int(num_digits))
            os.rename(filename+ext, "".join(arr)+ext)

 
 
if __name__ == "__main__":
    renseq()

Download this code: renseq.py

This works on all the files in the current directory. Pass an integer to renseq.py and it will change all the numbers in a filename (if there are any) to the same numbers, padded with zeros if they have fewer digits than the amount you want. So on the example

renseq.py 2

will turn the first list into the second list.

If say, there are filenames with numbers of three digits and you pass 2 to renseq.py, the numbers will be preserved (so it’s not a destructive rename), you’ll just revert to your incorrect ordering as it was in the beginning.

renseq.py will rewrite all t