When scripting is too damn slow

Israel Torres wrote a nice write-up on Google hacking to dig up some Wolfram Alpha API tags. This isn’t about that, but I thought the article was an interesting read. What this is about is something I noticed in his API generation script: it was damnably slow.

Now, before I go into my bit about this, let me emphasize: This is used only as an example to learn from; it is not a personal criticism of Israel in any way. Why put that in big bold letters? Because I’ve run into too many people who think they are smug and that writing something like this would be a way to prove personal superiority to somebody else. That’s not what this is about.

What is important is recognizing places where tight loops might need something a little lower level or faster. Israel’s code suffered a few things that can easily bite somebody: poor randomness from the shell’s $RANDOM and starting a ton of processes in a tight loop. In fact, the example scripting starts a new process (/usr/bin/printf) for every two bytes printed.

From his article:

Generating a million AppIDs takes under an hour on a modern system and validating them takes even longer (about 6 times longer). Interestingly, out of 1 million generated AppIDs only about 100K are unique; generating a true 1M unique IDs would take even longer! (See Figure 13 below)

Well, let’s check out Figure 15 from there instead as that’s the code:

That didn’t feel right to me on sight, so I tried a quick bit of coding:

The results were much more favorable when using C, and on a 3 year old MacBook laptop at that:

$ time ./a.out > /dev/null

real 0m1.337s
user 0m1.323s
sys 0m0.005s

For another 30 seconds of CPU work, I could pipe it to “sort | uniq | wc -l” and verify that I had a million unique entries.

In a first for me, I actually wrote a code snippet in C… then went back to my default go-to language for a speed comparison. Quickly cutting it into Perl, the code runs in 10 seconds.

In short, think about when your scripts might be slow on a production system, or when you’re generating a lot of data in a tight loop. It’s often best to write whatever gets the job done fastest overall. For the most part, whatever you can write quickly in works just fine, but sometimes remember performance — especially if what you wrote is going to spawn a lot of processes.

While I’m talking about “gotchas,” give yourself a pat on the back if you noticed what would be a huge mistake in anything involving cryptography: I didn’t seed my random number generator. The C code will generate the same list every time.

  1. Shell scripting is fast enough unless you call external binaries very often and get bottlenecked by invocation time. There are a few ways to workaround that.

    First, never run anything invocation-intensive in interactive shells because that decreases performance 10 to 100 times depending on the shell and its configuration.

    Second, use dash instead of bash. It has almost exactly the same feature set for a non-interactive shell, but it’s ~10 times faster.

    Third, try to use built-in functions instead of calling external binaries whenever possible. This avoids invoking another binary altogether.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>