Python Multiprocess diff between Windows and Linux

最新推荐文章于 2022-10-21 10:55:27 发布

转载最新推荐文章于 2022-10-21 10:55:27 发布 · 150 阅读

Python 专栏收录该内容

12 篇文章

订阅专栏

本文探讨了Python多进程模块在不同操作系统上的表现差异，特别是在Windows平台上处理全局变量的问题。作者通过修改代码解决了跨平台运行时的问题，并分享了解决方案。

I have a script called jobrunner.py that calls class methods in main.py. See below...

# jobrunner.py
from multiprocessing import Process
import main
from main import BBOX

def _a(arg):
    f = main.a()
    print f.run()

def _b(arg):
    p = main.b()
    print p.run()

if __name__ == '__main__':
    world = '-180,180,-90,90'
    BBOX.append(world.split(','))

    p1 = Process(target=_a, args=("1",))
    p2 = Process(target=_b, args=("1",))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

Processes _a and _b are invoked without any problems on OSX and Ubuntu but when I try to run the same thing on Windows (same version of python and all), it fails saying that index is out of range. This leads me to believe that the "global" variable BBOX is not being set or passed between modules on the Windows platform. Has anyone else seen something like this and know how to fix it?

Adam

UPDATE: I figured it out even though it might be a total hack...See below!

# jobrunner.py
from multiprocessing import Process
import main
from main import BBOX

def _a(arg):
    BBOX.append(arg) #This is the key
    f = main.a()
    print f.run()

def _b(arg):
    BBOX.append(arg) #This is the key
    p = main.b()
    print p.run()

if __name__ == '__main__':
    world = '-180,180,-90,90'
    BBOX.append(world.split(','))

    p1 = Process(target=_a, args=(BBOX[0],))
    p2 = Process(target=_b, args=(BBOX[0],))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

You shouldn't expect the values of global variables that you set in the parent process to be automatically propagated to the child processes.

Your code happens to work on Unix-like platforms because on those platforms multiprocessing uses fork(). This means that every child processes gets a copy of the parent process's address space, including all global variables.

This isn't the case on Windows; every variable from the parent process that needs to be accessed by the child has to be explicitly passed down or placed in shared memory.

Once you do this, your code will work on both Unix and Windows.

http://stackoverflow.com/questions/6596617/python-multiprocess-diff-between-windows-and-linux