[python] pool

本文深入探讨了Python中multiprocessing模块的使用方法,包括同步和异步执行任务的区别,以及如何利用Pool对象的apply、apply_async、map和map_async等方法进行高效并发编程。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

http://forlinux.blog.51cto.com/8001278/1423390

from multiprocessing import Pool

apply 开启多个进程并发执行

apply_async 同上,但是这个是异步的,非阻塞的。

map 类似于内建函数map,后面提供的参数列表会一个一个应用于函数,。这里会开发多个进程并发一起执行。

map_async 和map相同,只不过这是一个异步的,不会阻塞等待结果。该函数会返回一个结果对象。


https://docs.python.org/2/library/multiprocessing.html

apply(func[, args[, kwds]])

Equivalent of the apply() built-in function. It blocks until the result is ready, so apply_async() is better suited for performing work in parallel. Additionally, func is only executed in one of the workers of the pool.

apply_async(func[, args[, kwds[, callback]]])

A variant of the apply() method which returns a result object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed). callback should complete immediately since otherwise the thread which handles the results will get blocked.

map(func, iterable[, chunksize])

A parallel equivalent of the map() built-in function (it supports only one iterable argument though). It blocks until the result is ready.

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

map_async(func, iterable[, chunksize[, callback]])

A variant of the map() method which returns a result object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed). callback should complete immediately since otherwise the thread which handles the results will get blocked.

It blocks until the result is ready.

    import multiprocessing  
    import time  

    def f(x):  
        time.sleep(2)  
        print x  

    if __name__ == '__main__':   
      pool = multiprocessing.Pool(processes=5)   
      pool.map(f, xrange(10))  
      print("end")

暂时的理解是如果调用了map, 程序会阻塞在pool.map(f, xrange(10))这一行,直到它全部执行完才会 print("end")


http://heipark.iteye.com/blog/2081423
1.使用map方法

    import multiprocessing  
    import time  

    def f(x):  
        time.sleep(2)  
        print x  

    if __name__ == '__main__':   
      pool = multiprocessing.Pool(processes=5)   
      pool.map(f, xrange(10))  

并发5个进程
map方法会依次将参数二数组每个元素传入参数1方法中
如果将map()替换为map_async(),则方法不会阻塞,而是直接执行main进程后面的代码,所以要配合pool.close()和pool.join()一起使用。close()方法是使pool不再接受新任务;join()方法是阻塞main进程等待子进程执行完成才可以运行后面code

2.使用apply方法

    import multiprocessing  
    import time  

    def func(msg):  
        print msg  
        time.sleep(1)  

    if __name__ == "__main__":  
        print 'start main-process'  
        pool = multiprocessing.Pool(processes=4)  
        for i in xrange(10):  
            msg = "hello %d" %(i)  
            pool.apply_async(func, (msg, ))  
        pool.close()  
        pool.join()  
        print "Sub-process(es) done."  
        print 'Main-process done.'  

apply_async和apply方法区别是前者不会阻塞main进程,需要用pool.close()和join()方法阻塞等待子进程执行。
apply与map方法相比,它只是调用方法和参数,而map方法会将数组参数迭代传给被调用方法
pool.apply_async(func, (msg, )) 这行msg后面的逗号是不能省略的,否则不会执行func方法


http://stackoverflow.com/questions/8533318/python-multiprocessing-pool-when-to-use-apply-apply-async-or-map

Pool.apply blocks until the function is completed.

Pool.apply_async is also like Python’s built-in apply, except that the call returns immediately instead of waiting for the result. An ApplyResult object is returned. You call its get() method to retrieve the result of the function call. The get() method blocks until the function is completed. Thus, pool.apply(func, args, kwargs) is equivalent to pool.apply_async(func, args, kwargs).get().

In contrast to Pool.apply, the Pool.apply_async method also has a callback which, if supplied, is called when the function is complete. This can be used instead of calling get().

For example:

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool()
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

may yield a result such as

[1, 0, 4, 9, 25, 16, 49, 36, 81, 64]

Notice, unlike pool.map, the order of the results may not correspond to the order in which the pool.apply_async calls were made.

So, if you need to run a function in a separate process, but want the current process to block until that function returns, use Pool.apply. Like Pool.apply, Pool.map blocks until the complete result is returned.

If you want the Pool of worker processes to perform many function calls asynchronously, use Pool.apply_async. The order of the results is not guaranteed to be the same as the order of the calls to Pool.apply_async.

Notice also that you could call a number of different functions with Pool.apply_async (not all calls need to use the same function).

In contrast, Pool.map applies the same function to many arguments. However, unlike Pool.apply_async, the results are returned in an order corresponding to the order of the arguments.

Python的multiprocessing模块中的Pool类可以用来创建一个进程池,其中可以指定进程的数量。当有新的请求提交到Pool中时,如果池还没有满,就会创建一个新的进程来执行请求。如果池满,请求就会告知先等待,直到池中有进程结束,才会创建新的进程来执行这些请求。 使用Pool类的apply_async方法可以异步地向进程池提交任务。这个方法接受两个参数,第一个是要执行的函数,第二个是函数的参数。例如,我们可以定义一个函数func来打印数字,然后使用apply_async方法将任务提交到进程池中。在提交任务之前,我们需要调用close函数来关闭进程池的输入,然后再调用join函数来等待所有子进程执行完毕。 总结来说,PythonPool类提供了一种方便的方式来创建进程池,并且可以异步地向进程池中提交任务,从而实现多进程的并行执行。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* [python 进程池pool使用详解](https://blog.csdn.net/ayu6_1/article/details/124099288)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_2"}}] [.reference_item style="max-width: 50%"] - *2* *3* [Python学习笔记之进程池pool](https://blog.csdn.net/json_ligege/article/details/128084636)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_2"}}] [.reference_item style="max-width: 50%"] [ .reference_list ]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值