Python多进程并发(multiprocessing)

最新推荐文章于 2020-11-28 16:36:19 发布

转载最新推荐文章于 2020-11-28 16:36:19 发布 · 856 阅读

文章标签：

#Python multiprocessi

Python 专栏收录该内容

2 篇文章

订阅专栏

本文详细介绍了Python中多进程技术的实现方法，包括如何使用multiprocessing库创建单一进程、使用进程池并关注结果，以及如何在多核CPU上高效运行多个进程。通过实例演示了如何将单进程应用转换为并发执行的多进程应用。

由于Python设计的限制(我说的是咱们常用的CPython)。最多只能用满1个CPU核心。

Python提供了非常好用的多进程包multiprocessing，你只需要定义一个函数，Python会替你完成其他所有事情。借助这个包，可以轻松完成从单进程到并发执行的转换。

1、新建单一进程

import multiprocessing
import time

def func(msg):
    for i in xrange(3):
	    print msg
	    time.sleep(1)

if __name__ == "__main__":
    p = multiprocessing.Process(target=func, args=("hello", ))
    p.start()
    p.join()
    print "Sub-process done."

2、使用进程池

是的，你没有看错，不是线程池。它可以让你跑满多核CPU，而且使用方法非常简单。

注意要用apply_async，如果落下async，就变成阻塞版本了。

processes=4是最多并发进程数量。

import multiprocessing
import time

def func(msg):
    for i in xrange(3):
	    print msg
	    time.sleep(1)

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=4)
    for i in xrange(10):
        msg = "hello %d" %(i)
        pool.apply_async(func, (msg, ))
    pool.close()
    pool.join()
    print "Sub-process(es) done."

3、使用Pool，并需要关注结果

更多的时候，我们不仅需要多进程执行，还需要关注每个进程的执行结果，如下：

import multiprocessing
import time

def func(msg):
    for i in xrange(3):
	    print msg
	    time.sleep(1)
    return "done " + msg

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=4)
    result = []
    for i in xrange(10):
        msg = "hello %d" %(i)
        result.append(pool.apply_async(func, (msg, )))
    pool.close()
    pool.join()
    for res in result:
        print res.get()
    print "Sub-process(es) done."

Another example--

#! /usr/bin/env python
import random
import multiprocessing
import time
import Queue

class Worker(multiprocessing.Process):

    def __init__(self, work_queue, result_queue):

        # base class initialization
        multiprocessing.Process.__init__(self)

        # job management stuff
        self.work_queue = work_queue
        self.result_queue = result_queue
        self.kill_received = False

    def run(self):
        while not self.kill_received:

            # get a task
            #job = self.work_queue.get_nowait()
            try:
                job = self.work_queue.get_nowait()
            except Queue.Empty:
                break

            # the actual processing
            print("Starting " + str(job) + " ...")
            delay = random.randrange(1,3)
            time.sleep(delay)

            # store the result
            self.result_queue.put(delay)

if __name__ == "__main__":

    num_jobs = 20
    num_processes=8

    # run
    # load up work queue
    work_queue = multiprocessing.Queue()
    for job in range(num_jobs):
        work_queue.put(job)

    # create a queue to pass to workers to store the results
    result_queue = multiprocessing.Queue()

    # spawn workers
    for i in range(num_processes):
        worker = Worker(work_queue, result_queue)
        worker.start()

    # collect the results off the queue
    results = []
    for i in range(num_jobs):
        print(result_queue.get())