For some reason or another, I’m getting a lot of long-term hits to this post that I wrote a few months ago on parallel programming with Python. Thus, I thought I might share the news with people who have been looking for solutions to this problem: Python 2.6, which was just released, contains a new multiprocessing library which at first glance seems to formalize the procedure I outlined in my previous post and make it more robust. From the description:
multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.
If you’re interested in that, go grab 2.6 and check out the documentation for the package here!