New parallel programming tool in Python 2.6

For some reason or another, I’m getting a lot of long-term hits to this post that I wrote a few months ago on parallel programming with Python.  Thus, I thought I might share the news with people who have been looking for solutions to this problem:  Python 2.6, which was just released, contains a new multiprocessing library which at first glance seems to formalize the procedure I outlined in my previous post and make it more robust.  From the description:

multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

If you’re interested in that, go grab 2.6 and check out the documentation for the package here!


5 Responses to New parallel programming tool in Python 2.6

  1. M_D_K says:

    Pretty cool, it sounds like there using microthreads like in stackless python.

  2. Rhamphoryncus says:

    Nope, multiprocessing uses processes, as the name says.

  3. Jennifer says:

    You said “For some reason or another…” and I think the reason is, in fact, the release of python 2.6 and people’s use of the multiprocessing module of which you speak.

    The trick is that processes are hard to abstract over in a platform agnostic way and the code seems (maybe?) to have lurking bugs due to a (possibly?) rushed delivery.

    I think “people” are running into problems using multiprocessing and then googling around for solutions or bug reports or *something* to explain what’s going wrong. Anyway, that’s how I found your other article and subsequently this one by way of trackback 🙂

    For reference, the problem I’m dealing with is that my script fails to die gracefully. When I ^C out of it, it is very frequently in multiprocessing.forking.poll() sitting on the line “pid, sts = os.waitpid(, flag)”. I found your previous post when I started broadening my google searches around python fork() topics based on this observation.

  4. Jennifer says:

    To help anyone else who stumbles across this blog post, the thing that ended up solving my problem was using multiprocessing.Manager() and getting a manager.Queue() from it, instead of using a plain old multiprocessing.Queue().

    The manager objects are rarely in any online examples I’ve found. They tend to demo the basic functionality with toy problems like having a process print its id or some such. The value of the Manager shows up in preventing some sort of deadlocking problem that pops up sporadically when passing data between processes.

  5. Dannielle says:

    I do not know if it’s just me orr if perhaps everybody else encountering issues with your
    site. It appears like some of the text iin your
    posts are running off the screen. Cann someone else please provide feedback
    and let me know if this is happening to them too?

    This might bbe a problem with my internet brrowser because I’ve had this happen before.
    Appreciate it

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s