Using exceptions for goto

Goto is a shunned construct in modern programming languages, but occasionally there is a case where it makes sense, such as breaking-out from a set of nested for statements when a solution is found. But, modern programming languages contain a better construct for such cases---exceptions. For example, say we are trying to find an item from each of three sets which jointly satisfy some criterion. A naive implementation might look like:

foundMatch = False
for item1 in set1:
    for item2 in set2:
        for item3 in set3:
            if satisfiesCriterion(item1, item2, item3):
                foundMatch = True
                break
        if foundMatch:
            break
    if foundMatch:
        break

But, this code can be simplified by introducing an exception:

try:
    for item1 in set1:
        for item2 in set2:
            for item3 in set3:
                if satisfiesCriterion(item1, item2, item3):
                    raise FoundMatch()
except FoundMatch:
    pass

Twisted Documentation

There is currently much discussion on the twisted mailing list about improving twisted documentation. I'm one of many who think the documentation could be improved. I found a major problem to be a lack of introduction to the twisted mental model---the fact that it uses cooperative timesharing and blocking calls to handle events.

Victor Norman suggested Dave Peticolas' Twisted Introduction. Reading the first article which explains the Twisted "mental model" felt like a breath of fresh air. I disagree with his use of asynchronous, which implies parallel, non-blocking, etc. But, starting with the mental model is definitely the right approach. Now, if only this documentation could be integrated with the main documentation...

P.S. Dave Peticolas---I've heard that name before. Sure enough, he worked on GnuCash, my accounting program of choice.

Twisted: callWhenRunning, callFromThread or callLater?

When I first learned of reactor.callWhenRunning, I apparently didn't read the documentation and/or source code sufficiently carefully. I correctly understood that it was the function to use when you wanted to queue a function to be called immediately after reactor start. My mistake was to believe that it queued the function if the reactor had already been started. In fact, if the reactor is in the "running" state, it simply calls the specified function. I wonder if part of the reason for this design is how it handles the not-running case. If the reactor is not running, callWhenRunning adds a startup trigger for the specified function. Such a trigger cannot be used to queue-up a task/call.

I learned (the hard way) of the need for callFromThread when trying to run a web server and twisted reactor in separate threads of the same process ("don't try this at home"). Jean-Paul's answer to my question about reactor.wakeUp provides the reason for this requirement. The reactor must make blocking calls (e.g. select()) for certain functionality (e.g. networking). The wakeUp trips the blocking call by, e.g., "writ[ing] a byte to a pipe the reactor is select()ing (etc) on". In my case, I found that an attempt by the web server code to write to the network might be ignored indefinitely unless the call was wrapped with callFromThread. What does callFromThread do? It adds the function to the threadCallQueue and "wakes up" the reactor. Unlike callWhenRunning the specified function call isn't made until after callFromThread returns, so it can be used to queue-up a function for running when the reactor (re-)gains control.

If you read the callFromThread documentation, you'll find that callLater is the recommended way (with delay=0) to queue a function for calling in the next mainLoop iteration. Like callFromThread, callLater uses a queue(s) to manage the calls. Two queues are kept: one for calls which haven't waited long enough (_newTimedCalls), and one for calls which have waited long enough, but haven't been called yet (_pendingTimedCalls). The _pendingTimedCalls are called during the next mainLoop iteration.

Half-closing a TCP connection in Twisted

loseWriteConnection is the function I had been looking for all day. In retrospect, it was obvious---just look at the ITCPTransport manual page. But, at first I didn't know what I was looking for---I was just confused as to why netcat wasn't working as expected.

I was trying to get server status information which required sending a simple command to the server. When I used a custom netcat-like utility, it worked, but when I used netcat or python/twisted, it didn't. At first, I thought the special utility might have been sending an extra EOF-like character, but some testing eliminated that possibility. Then, I thought it might be a feed-line issue. Nope. Finally, I realized the problem---netcat and python/twisted weren't half-closing the write connection after sending the command. How did I come to this conclusion? I tried the netcat -q option and immediately got back the server status information (before the specified timeout).

Earlier, I had tried to (half-)close the connection with python/twisted using ITransport.loseConnection. But, after fully realizing the half-close issue and making additional loseConnection attempts, I concluded that loseConnection fully closes the connection, losing the response. Next, I found _closeWriteConnection which sounded like it would do exactly what I wanted. The source even looked like it would work, but for whatever reason it didn't. Finally, I was clued-into loseWriteConnection which closed the write-side of the connection while still allowing reading of the server response.

Running Tests

For a python project I worked on, we used the standard python unittest module and placed test classes within an if __name__=='__main__': block at the bottom of each module. This makes tests easy to run and has the advantage of keeping the testing code close to the source code. But, as I've learned, there's a better way to do it.

The major drawback of the above framework is a lack of control over tests. One cannot selectively run tests from within a module nor can test results be compiled in a nice way (since screen-scraping is the only option). I've since learned about nose, which is a "test runner." Instead of wrapping unit test classes in a if __name__=='__main__':, you simply place test classes somewhere in your source code hierarchy. Options include along-side the module code, or in "test" files within a "test" directory. To run tests, you simply run nosetests with arguments specifying what tests you want to run. This could be the root directory of your source code tree, or a list of python module names. Further refinement of which tests to run can be had by using nose attributes.

An absolutely relative import

Part of the "What's New" documentation for python 2.5 describes how to make use of absolute imports. After reading this, you might find the following example confusing. I sure was confused after trying it.

Create string.py:

import string
a = 1

Create main.py:

from __future__ import absolute_import
import string
print string.a

Both scripts should be placed in the same directory. Run main.py:

$ python main.py

You'll see main.py print "1", the value set by string.py. A reading of the python documentation might lead you to believe that this behavior is incorrect---it should instead import the standard library string module and raise an AttributeError. This interpretation is correct except for that, by default, python includes the script directory in the list of "absolute" import paths. So, the easy fix is to delete this entry which conveniently is always found at the beginning of sys.path. The revised main.py is:

from __future__ import absolute_import
import sys
sys.path = sys.path[1:]
import string
print string.a

I appreciate that python has moved to a cleaner import system. But, leaving the script/current directory in the list of "absolute" import paths seems like a huge oversight.

What's especially ridiculous about the default behavior is that if you have a module with the same name as a standard library module, import the standard library module, and include unittests at the bottom, the unittests won't work because the import will behave differently depending on whether the module is imported or run as a script. This is the problem that initially brought me down this path...

Update 9/23: After talking with different people about this issue, I've learned that it's easy to think that sys.path.remove('.') is the right thing to do here. It's not. The default local path inserted by python may be a full path or an empty string in which case sys.path.remove('.') won't fix the problem. Trying to remove all local directory entries is also incorrect since the user may genuinely want to include the local directory in the search path.

jsonlib

For a project I worked on at ITA, we decided to use pickle for internal object serialization/communication. Pickle certainly makes coding simple, but I've occasionally wondered whether we made the best choice. I found this article comparing deserialization libraries to be interesting. It sounds like the two main competing camps are json and Google's protocol buffers. It sounds like protocol buffers is slow (in python) because it is pure python and not optimized for speed. One json library, jsonlib sounds like the right way to go as it provides faster speeds and more compact storage than pickle.