Python Quirks

Twisted: Asynchronous HTTP Request

2011-04-02T20:56:00.001-04:00

Note that how to make an HTTP request with Twisted is already documented. But, unless you're already familiar with Twisted, my guess is that extending the example code to downloading a large number of web pages with a limit on the number of simultaneous requests is not easy. Below, you'll find example code for exactly that. Below the code is a walk-through that will hopefully help you understand the details.


from pprint import pformat

from twisted.internet import reactor
import twisted.internet.defer
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers

class PrinterClient(Protocol):
    def __init__(self, whenFinished):
        self.whenFinished = whenFinished

    def dataReceived(self, bytes):
        print '##### Received #####\n%s' % (bytes,)

    def connectionLost(self, reason):
        print 'Finished:', reason.getErrorMessage()
        self.whenFinished.callback(None)

def handleResponse(r):
    print "version=%s\ncode=%s\nphrase='%s'" % (r.version, r.code, r.phrase)
    for k, v in r.headers.getAllRawHeaders():
        print "%s: %s" % (k, '\n  '.join(v))
    whenFinished = twisted.internet.defer.Deferred()
    r.deliverBody(PrinterClient(whenFinished))
    return whenFinished

def handleError(reason):
    reason.printTraceback()
    reactor.stop()

def getPage(url):
    print "Requesting %s" % (url,)
    d = Agent(reactor).request('GET', url, Headers({'User-Agent': 'twisted']}), None)
    d.addCallbacks(handleResponse, handleError)
    return d

semaphore = twisted.internet.defer.DeferredSemaphore(2)
dl = list()
dl.append(semaphore.run(getPage, 'http://google.com'))
dl.append(semaphore.run(getPage, 'http://cnn.com'))
dl.append(semaphore.run(getPage, 'http://nytimes.com'))
dl = twisted.internet.defer.DeferredList(dl)
dl.addCallbacks(lambda x: reactor.stop(), handleError)

reactor.run()

getPage handles an entire single HTTP request. Agent(reactor).request() creates an Agent and sends the HTTP request. request() returns a deferred which is fired when the headers are retrieved. The addCallbacks line specifies that handleResponse is called upon successful header retrieval and handleError is called if there is an error in retrieving the headers.

handleResponse is given a Response object which contains the HTTP header and includes a method, deliverBody, to specify a Protocol to handle delivery of the HTTP body. A Protocol is used for body delivery because it may come in chunks and an error may occur in the middle of delivery (e.g. someone pulls your network plug). PrinterClient is a very simple Protocol which (1) prints received data, (2) logs the reason for termination (if not twisted.web.client.ResponseDone, there was an error), and (3) fires a deferred whenFinished.

The trickiest part of this code is following the Deferred chain, which is essential to understanding how we limit the maximum number of outstanding requests. A key point to understand about Deferreds is that, if a callback returns a Deferred, the parent Deferred waits for the child Deferred to fire before handing a value to the next Deferred in the chain. See documentation on Chaining Deferreds. Because of this, each semaphore.run waits for the PrinterClient protocol to complete before releasing its semaphore. The DeferredSemaphore is basically a Deferred-aware semaphore. It's only argument is the number of tokens it allows to be "checked-out" simultaneously. When we make the nytimes.com semaphore.run call, the semaphore doesn't call getPage until one of the other requests has completed.

The DeferredList is used to clean-up after all requests have completed. Under normal circumstances, we just want to stop the reactor so our process will exit. But, if there is an error, we want to see what happened, hence we use handleError in that case.

Update 9/13/11: Minor code formatting change.

Twisted: Beware: Returning a Value from dataReceived

2011-03-09T14:18:00.004-05:00

We just lost approximately 10-man-hours to undocumented behavior of Twisted. If you return a True truth value from your dataReceived function (after it is called by the reactor), the reactor will destroy your protocol, and close the corresponding connection. Fortunately, this behavior is recognized as a bug and a deprecation warning will be likely be issued with this behavior in the 11.0 release. But, since many of us are stuck with Twisted 10.2 or earlier for months, if not years, to come, it's good to be aware of this issue.

If you Misspell "protocol", this is what you get

2011-02-25T15:55:00.001-05:00

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 51, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 36, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
---  ---
  File "/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py", line 563, in doConnect
    self._connectDone()
  File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py", line 566, in _connectDone
    self.protocol = self.connector.buildProtocol(self.getPeer())
  File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 930, in buildProtocol
    return self.factory.buildProtocol(addr)
  File "/usr/lib/python2.5/site-packages/twisted/internet/protocol.py", line 98, in buildProtocol
    p = self.protocol()

This is worth remembering. Note that nothing here refers to the corresponding factory or the code where the error was made. I think this is one reason Twisted can be frustrating.

You can see this error with an simple example:

from twisted.internet import protocol
from twisted.internet import reactor
class MyProtocol(protocol.Protocol):
    pass
class MyFactory(protocol.ReconnectingClientFactory):
    protcol = MyProtocol
reactor.connectTCP('google.com', 80, MyFactory())
reactor.run()

It's certainly convenient to be able to set the protocol so simply, but it's disappointing that the error isn't caught at the source. I wonder why the factories don't have an __init__ method that checks for a valid protocol field?

Using exceptions for goto

2011-01-26T11:39:00.001-05:00

Goto is a shunned construct in modern programming languages, but occasionally there is a case where it makes sense, such as breaking-out from a set of nested for statements when a solution is found. But, modern programming languages contain a better construct for such cases---exceptions. For example, say we are trying to find an item from each of three sets which jointly satisfy some criterion. A naive implementation might look like:

foundMatch = False
for item1 in set1:
    for item2 in set2:
        for item3 in set3:
            if satisfiesCriterion(item1, item2, item3):
                foundMatch = True
                break
        if foundMatch:
            break
    if foundMatch:
        break

But, this code can be simplified by introducing an exception:

try:
    for item1 in set1:
        for item2 in set2:
            for item3 in set3:
                if satisfiesCriterion(item1, item2, item3):
                    raise FoundMatch()
except FoundMatch:
    pass

Twisted Documentation

2011-01-20T10:00:00.003-05:00

There is currently much discussion on the twisted mailing list about improving twisted documentation. I'm one of many who think the documentation could be improved. I found a major problem to be a lack of introduction to the twisted mental model---the fact that it uses cooperative timesharing and blocking calls to handle events.

Victor Norman suggested Dave Peticolas' Twisted Introduction. Reading the first article which explains the Twisted "mental model" felt like a breath of fresh air. I disagree with his use of asynchronous, which implies parallel, non-blocking, etc. But, starting with the mental model is definitely the right approach. Now, if only this documentation could be integrated with the main documentation...

P.S. Dave Peticolas---I've heard that name before. Sure enough, he worked on GnuCash, my accounting program of choice.

Twisted: callWhenRunning, callFromThread or callLater?

2011-01-12T16:01:00.001-05:00

When I first learned of reactor.callWhenRunning, I apparently didn't read the documentation and/or source code sufficiently carefully. I correctly understood that it was the function to use when you wanted to queue a function to be called immediately after reactor start. My mistake was to believe that it queued the function if the reactor had already been started. In fact, if the reactor is in the "running" state, it simply calls the specified function. I wonder if part of the reason for this design is how it handles the not-running case. If the reactor is not running, callWhenRunning adds a startup trigger for the specified function. Such a trigger cannot be used to queue-up a task/call.

I learned (the hard way) of the need for callFromThread when trying to run a web server and twisted reactor in separate threads of the same process ("don't try this at home"). Jean-Paul's answer to my question about reactor.wakeUp provides the reason for this requirement. The reactor must make blocking calls (e.g. select()) for certain functionality (e.g. networking). The wakeUp trips the blocking call by, e.g., "writ[ing] a byte to a pipe the reactor is select()ing (etc) on". In my case, I found that an attempt by the web server code to write to the network might be ignored indefinitely unless the call was wrapped with callFromThread. What does callFromThread do? It adds the function to the threadCallQueue and "wakes up" the reactor. Unlike callWhenRunning the specified function call isn't made until after callFromThread returns, so it can be used to queue-up a function for running when the reactor (re-)gains control.

If you read the callFromThread documentation, you'll find that callLater is the recommended way (with delay=0) to queue a function for calling in the next mainLoop iteration. Like callFromThread, callLater uses a queue(s) to manage the calls. Two queues are kept: one for calls which haven't waited long enough (_newTimedCalls), and one for calls which have waited long enough, but haven't been called yet (_pendingTimedCalls). The _pendingTimedCalls are called during the next mainLoop iteration.

Half-closing a TCP connection in Twisted

2010-12-01T15:21:00.005-05:00

loseWriteConnection is the function I had been looking for all day. In retrospect, it was obvious---just look at the ITCPTransport manual page. But, at first I didn't know what I was looking for---I was just confused as to why netcat wasn't working as expected.

I was trying to get server status information which required sending a simple command to the server. When I used a custom netcat-like utility, it worked, but when I used netcat or python/twisted, it didn't. At first, I thought the special utility might have been sending an extra EOF-like character, but some testing eliminated that possibility. Then, I thought it might be a feed-line issue. Nope. Finally, I realized the problem---netcat and python/twisted weren't half-closing the write connection after sending the command. How did I come to this conclusion? I tried the netcat -q option and immediately got back the server status information (before the specified timeout).

Earlier, I had tried to (half-)close the connection with python/twisted using ITransport.loseConnection. But, after fully realizing the half-close issue and making additional loseConnection attempts, I concluded that loseConnection fully closes the connection, losing the response. Next, I found _closeWriteConnection which sounded like it would do exactly what I wanted. The source even looked like it would work, but for whatever reason it didn't. Finally, I was clued-into loseWriteConnection which closed the write-side of the connection while still allowing reading of the server response.

Running Tests

2010-09-24T08:42:00.000-04:00

For a python project I worked on, we used the standard python unittest module and placed test classes within an if __name__=='__main__': block at the bottom of each module. This makes tests easy to run and has the advantage of keeping the testing code close to the source code. But, as I've learned, there's a better way to do it.

The major drawback of the above framework is a lack of control over tests. One cannot selectively run tests from within a module nor can test results be compiled in a nice way (since screen-scraping is the only option). I've since learned about nose, which is a "test runner." Instead of wrapping unit test classes in a if __name__=='__main__':, you simply place test classes somewhere in your source code hierarchy. Options include along-side the module code, or in "test" files within a "test" directory. To run tests, you simply run nosetests with arguments specifying what tests you want to run. This could be the root directory of your source code tree, or a list of python module names. Further refinement of which tests to run can be had by using nose attributes.

An absolutely relative import

2010-07-13T08:17:00.008-04:00

Part of the "What's New" documentation for python 2.5 describes how to make use of absolute imports. After reading this, you might find the following example confusing. I sure was confused after trying it.

Create string.py:

import string
a = 1

Create main.py:

from __future__ import absolute_import
import string
print string.a

Both scripts should be placed in the same directory. Run main.py:

$ python main.py

You'll see main.py print "1", the value set by string.py. A reading of the python documentation might lead you to believe that this behavior is incorrect---it should instead import the standard library string module and raise an AttributeError. This interpretation is correct except for that, by default, python includes the script directory in the list of "absolute" import paths. So, the easy fix is to delete this entry which conveniently is always found at the beginning of sys.path. The revised main.py is:

from __future__ import absolute_import
import sys
sys.path = sys.path[1:]
import string
print string.a

I appreciate that python has moved to a cleaner import system. But, leaving the script/current directory in the list of "absolute" import paths seems like a huge oversight.

What's especially ridiculous about the default behavior is that if you have a module with the same name as a standard library module, import the standard library module, and include unittests at the bottom, the unittests won't work because the import will behave differently depending on whether the module is imported or run as a script. This is the problem that initially brought me down this path...

Update 9/23: After talking with different people about this issue, I've learned that it's easy to think that sys.path.remove('.') is the right thing to do here. It's not. The default local path inserted by python may be a full path or an empty string in which case sys.path.remove('.') won't fix the problem. Trying to remove all local directory entries is also incorrect since the user may genuinely want to include the local directory in the search path.

jsonlib

2010-07-07T18:37:00.000-04:00

For a project I worked on at ITA, we decided to use pickle for internal object serialization/communication. Pickle certainly makes coding simple, but I've occasionally wondered whether we made the best choice. I found this article comparing deserialization libraries to be interesting. It sounds like the two main competing camps are json and Google's protocol buffers. It sounds like protocol buffers is slow (in python) because it is pure python and not optimized for speed. One json library, jsonlib sounds like the right way to go as it provides faster speeds and more compact storage than pickle.

Returning an exit status with Twisted

2010-06-08T14:08:00.002-04:00

When I had a need for returning an exit status from a Twisted process, my first instinct was to look for a reactor.stop argument. In fact, there have been multiple requests for such, e.g. tickets #718 and #2182. But, then, I realized that reactor.stop doesn't stop the reactor, it merely initiates the shutdown process. The reactor is not shut down until reactor.run exits. This realization made it clear what I should do to return a specific exit code---simply add

    sys.exit(code)

immediately after reactor.run.

More ElementTree Annoyances

2010-03-22T10:30:00.003-04:00

Cannot serialize int. I can see the value in not automatically serializing every possible object with a __str__ method. But, not converting an int? C'mon!
Cannot serilaize None. Wouldn't None be the perfect value to indicate "don't serialize this attribute"?

I'm generally a fail-fast-and-loudly kind of guy, but I also don't like having to write more code when it's obvious what I mean. These seem like two cases where I think the tradeoff is in favor of writing less code...

Examples:

>>> import xml.etree.ElementTree as et
>>> et.tostring(et.Element('Foo', attrib={ 'a': 1}))
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 1009, in tostring
    ElementTree(element).write(file, encoding)
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 663, in write
    self._write(file, self._root, encoding, {})
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 698, in _write
    _escape_attrib(v, encoding)))
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 830, in _escape_attrib
    _raise_serialization_error(text)
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 777, in _raise_serialization_error
    "cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize 1 (type int)
>>> et.tostring(et.Element('Foo', attrib={ 'a': None}))
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 1009, in tostring
    ElementTree(element).write(file, encoding)
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 663, in write
    self._write(file, self._root, encoding, {})
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 698, in _write
    _escape_attrib(v, encoding)))
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 830, in _escape_attrib
    _raise_serialization_error(text)
  File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 777, in _raise_serialization_error
    "cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize None (type NoneType)

missing

2010-02-08T17:10:00.003-05:00

According to the python documentation:

If a subclass of dict defines a method __missing__(), if the key key is not present, the d[key] operation calls that method with the key key as argument. The d[key] operation then returns or raises whatever is returned or raised by the __missing__(key) call if the key is not present. No other operations or methods invoke __missing__(). If __missing__() is not defined, KeyError is raised. __missing__() must be a method; it cannot be an instance variable. For an example, see collections.defaultdict.

This is, at least, incomplete, since __missing__ must not only return the default value, but also assign it internally. This is made clear in the documentation for collections.defaultdict:

If default_factory is not None, it is called without arguments to provide a default value for the given key, this value is inserted in the dictionary for the key, and returned.

Surprisingly, the __missing__ method is not mentioned in the special method names section of the python documentation.

collections.defaultdict

2010-02-04T11:05:00.007-05:00

collections.defaultdict is nice, especially when counting things. But, defaultdict only lets you use zero-argument constructors. Pffft! Fortunately, it's easy to write a defaultdict which passes arguments to the constructor:

class defaultdict2(dict):
    def __init__(self, factory, factArgs=(), dictArgs=()):
        dict.__init__(self, *dictArgs)
        self.factory = factory
        self.factArgs = factArgs
    def __missing__(self, key):
        self[key] = self.factory(*self.factArgs)
        return self[key]

Update 2/8/10: added "return" line to __missing__ per discussion in this post on __missing__.

Kid Template Recompilation

2010-02-03T15:29:00.007-05:00

I'm involved in a project which uses the TurboGears framework for serving web pages. The templating language we use is Kid. Recently, we ran into a problem where web pages did not correspond to the installed templates. After a bit of detective work, we suspected that TurboGears/Kid was not using the templates, but rather stale, compiled versions of old templates (.pyc files). Some Kid mailing list discussion confirmed our suspicions. The problem is that Kid only recompiles if the mtime of the source (.kid) file is after the mtime of the corresponding compiled (.pyc) file. In contrast, Python recompiles unless the mtime stored in the .pyc file exactly matches the mtime of the source (.py) file.

My understanding is that, ideally, Python would use a one-way hash of the source and only use the compiled file if there is an exact match. The exact mtime comparison is practically nearly as good and much, much faster. But, the mtime inequality comparison is a poor approximation of the ideal and only works when you can guarantee that (1) the system clock is perfect and never changes timezone (e.g. no switch between EDT and EST), and (2) mtimes are always updated to "now" whenever contents or locations are changed (i.e. even "mv" must affect mtime and rsync -a is right out). I don't know of any OS which provides these guarantees. The good news is that there is no disagreement on the existence of the problem; so, this is likely to be fixed in a future version of Kid.

numpy.dot

2010-01-19T13:22:00.002-05:00

I should have known. numpy.dot doesn't work with sparse matrices. What's worse is that it happily accepts a sparse matrix as an argument and yields some convoluted array of sparse matrices. What I should be doing is x.dot(y) where x is a scipy.sparse.sparse.spmatrix and y is a numpy.ndarray.

Note that I'm using the Debian stable versions of these packages: numpy 1.1.0 and scipy 0.6.0.

urllib2.HTTPErrorProcessor

2010-01-08T11:51:00.005-05:00

With code similar to that I posed in Asynchronous HTTP Request, I was occasionally getting empty responses to my requests. When I added urllib2.HTTPErrorProcessor to the inheritance list for MyHandler, the problem went away. My guess is the server was generating a 503 Service Unavailable responses and my client code wasn't handling it. How one was supposed to know to do this from the documentation, I am unsure. I'm guessing that if the server might provide a redirect for your url, you would also want to inherit from urllib2.HTTPRedirectHandler.

Element.text and other ElementTree Annoyances

2009-12-28T11:58:00.013-05:00

I have a love/hate relationship with ElementTree. It generally makes processing and generating XML very easy. But, some of the design decisions feel like they were meant to frustrate, rather than help, the programmer:

Many __str__ and __repr__ methods return near-useless strings like <Element ElementName at 7fb1d0f63e60>. Would methods that specify attributes and text/tail properties really be so difficult to define? Even "def __str__(self): return tostring(self)" would be an improvement.
Element() cannot specify text. The Element factory only lets you specify the tag name and attributes. There is no argument you can pass to specify the text or tail. See, for example, a discussion about setting the text property. I see no point in forcing the programmer to write the extra line of code.
Interfaces. ElementTree hides the Element class and provides a factory which returns an object which implements the _ElementInterface. There are other languages in which I can see this being a useful practice. But, python does not have sufficient language support and I find that this half-hearted attempt at abstraction simply makes the module more difficult to use. Python already provides plenty of tools to hide "magic" which don't interrupt programmer intuitions. Why not use those?

Asynchronous HTTP Request

2009-12-23T15:43:00.015-05:00

Note (4/2/11): Please see my recent post detailing asynchronous HTTP requests using Twisted.

Note (3/13/11): I originally wrote this post while looking for callback-style HTTP request functionality in python. I made the mistake of thinking that "callback-style" is the same as "asynchronous". The following details my efforts to achieve a callback-style HTTP request using urllib2. The final (updated) code example illustrates how to use threads to achieve asynchronicity. I'd recommend using a thread pool if you plan more than just a handful of requests. And, as others have noted, Twisted is really the best python framework for asynchronous programming. Also, I'd like to thank the commenters for pointing out my mistakes; I'm sorry for not realizing my errors sooner.

You might think it would be easy to write python code to ~~perform an asynchronous~~ achieve a callback-style web request. It ought to be as simple as providing a url and callback function to some python library routine, no? Well, technically, it is that simple. But somehow, the documentation makes the task surprisingly difficult.

One option, of course, is Twisted. But, reading through the (sparse, fractured) documentation made me think there had to be something easier. This led me to urllib2. The short answer is that, yes, urllib2 does what I want. But, the documentation is sufficiently backwards that it took me over an hour to figure out how to accomplish the task.

Accomplishing a ~~blocking~~ simple HTTP request with urllib2 is simple and the documentation reflects that: use openurl. The return value of openurl provides the response and additional information in a file-like object. The problem is how to achieve the same result in an ~~asynchronous~~ callback-style manner. One would think openurl could simply take an additional handler object which is called with the response as its only argument when the request completes. Ha! build_opener looked vaguely promising as it accepted handler(s). This led me to create a class which inherited from BaseHandler which defined protocol_response. No dice. And, as I later realized, protocol_response takes three arguments (self, req, response), not two, and changes names depending on the protocol. Of course, at that point, I was at a loss as to how the protocol name was determined (the BaseHandler documentation ignored this issue). And, the examples were useless since they all used standard handlers. Next, I tried inheriting from HTTPHandler, overriding http_response with a method that simply prints the url, info and response text. This almost worked. It successfully retrieved the web page and printed it. But, then, it raised the following exception:

Traceback (most recent call last):
  File "./webtest.py", line 14, in 
    o.open('http://www.google.com/')
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
    response = meth(req, response)
  File "/usr/lib/python2.6/urllib2.py", line 496, in http_response
    code, msg, hdrs = response.code, response.msg, response.info()
AttributeError: 'NoneType' object has no attribute 'code'

After much searching, I finally realized that I had failed to return a response-like object from my http_response method. This seems like an odd requirement for a callback method. And, it could have been easily clarified in the documentation with an example.

Alas, after all that, I was able to use urllib2 to successfully make an asynchronous HTTP request, so I can't complain too much. Here's the code for anyone who's interested:

#!/usr/bin/env python

import urllib2
import threading

class MyHandler(urllib2.HTTPHandler):
    def http_response(self, req, response):
        print "url: %s" % (response.geturl(),)
        print "info: %s" % (response.info(),)
        for l in response:
            print l
        return response

o = urllib2.build_opener(MyHandler())
t = threading.Thread(target=o.open, args=('http://www.google.com/',))
t.start()
print "I'm asynchronous!"

Update (3/12/11): My comment before the sample code indicated that the sample code was asynchronous. But, it wasn't. I've updated it to be asynchronous. When originally writing this post, I intended the example code to show the urllib2 handler approach.

Reworking the GIL

2009-12-17T10:00:00.002-05:00

The title of this post is stolen from an email which describes steps the author has made to address GIL issues David Beazley raised with his talk on the Global Interpreter Lock. The proposed changes certainly won't turn Python into a completely thread-friendly language (the GIL is not going away any time soon), but it sounds like these changes will greatly reduce thread overhead and give the effect of running on a single-core machine that one would expect with a global interpreter lock.

Twisted Annoyances

2009-12-03T12:50:00.003-05:00

While Twisted is generally an excellent network library, it certainly has its quirks.

Exception Trapping: by default, the reactor will trap all exceptions. See ReactorBase.runUntilCurrent for the code that implements this horrible behavior. This breaks intuitions most developers have for how exceptions are supposed to work. For example, the python tutorial says:
The last except clause may omit the exception name(s), to serve as a wildcard. Use this with extreme caution, since it is easy to mask a real programming error in this way!
Twisted does an excellent job of masking real programming errors. I'm surprised that there is no option to turn off this behavior.
callFromThread: calling a reactor method (e.g. protocol.writeData) from a non-reactor thread doesn't work without wrapping it with this method.

Python Coil

2009-11-17T18:51:00.005-05:00

Coil is a nice configuration language for Python ~~created by Michael Marineau~~ created by Itamar Turner-Trauring and currently maintained by Michael Marineau. It is used here at ITA. It is much less verbose than XML but is very readable and minimizes duplication. I really wish there were a Debian package for it!

Update (11/18/09): Michael M. informed me of the correct history :)

Block Scoping

2009-10-28T13:46:00.007-04:00

Unlike languages like Java, C++ and Perl, Python does not have block scoping. I.e. if you define a variable inside a loop in Python, it will still be in scope after that loop and will override previous bindings to that name. Python instead delimits scope at the levels of module, class and function. I defer to the authoritative source for the gory details. Note that according to the definitions used therein, Python does have block-level scoping, but that is only if you define block delimiters to be modules, classes and functions :-)

Python scoping is a drawback in the sense that if you are used to (Java/C++/Perl) block scoping, you are likely to accidentally introduce bugs as a result of using the same variable at different block levels. I've introduced a few such bugs. On the other hand, Python scoping eliminates the need to pre-define/declare variables which are set in a loop/if block, yet you need access to afterward. So, even though I've been burned by this style of scoping, I find that it helps me write better (i.e. more readable) code.

The Global Interpreter Lock

2009-10-27T15:12:00.003-04:00

Python technically has threading capabilities. And, it can work quite well if the threads are i/o-bound. However, Python threading doesn't work so well when threads are cpu-bound. The following hour-long video explains why. Read the slides if you are impatient.

http://blip.tv/file/2232410

One observation that David Beazley makes is that only the "main" thread can deal with signals like Control-C. However, if this thread is blocked via a join(), the signal will not get handled. So, it may be worth creating a thread separate from the "main" thread to spawn and join threads. Haven't yet tested this, though...

Controlling Printing in Numpy

2009-10-19T17:13:00.000-04:00

Numpy has numpy.set_printoptions for controlling the printing of arrays. See the doc for full details. Now that I know about it, I'll be using something displaying fewer precision digits, allowing a larger linewidth and not summarizing until the array is substantially larger:

numpy.set_printoptions(precision=4,
                       threshold=10000,
                       linewidth=150)