分布式进程

分布式进程指的是将Process进程分布的多台机器上，充分利用多台机器的性能完成复杂的任务，我们可以将这点应用的分布式爬虫的开发中。

我们举个简单例子，服务进程用来设置任务在task_queue，并设置接口。任务进程调用相同的接口，执行任务，结果写进result queue

 

 

 

taskManager.py ： 服务进程

from multiprocessing.managers import BaseManager
from multiprocessing import freeze_support
import queue
#任务个数
task_number = 10
#定义收发队列
task_queue = queue.Queue(task_number)
result_queue = queue.Queue(task_number)

def get_task():
    return task_queue

def get_result():
    return result_queue

#创建类似的QueenManager
class QueueManager(BaseManager):
    pass
def win_run():
    QueueManager.register('get_task_queue',callable=get_task)
    QueueManager.register('get_result_queue',callable=get_result)
    manager = QueueManager(address = ('127.0.0.1',8001),authkey = 'qiye')
    manager.start()
    try:
        task = manager.get_task_queue()
        result = manager.get_result_queue()
        for url in ["ImageUrl_" + str(i) for i in range(10)]:
            print('put task %s....' % url)
            task.put(url)
        print('try get result....')
        for i in range(10):
            print('result is %s' % result.get(timeout=10))
    except:
        print('Manager error')
    finally:
        manager.shutdown()


if __name__ == '__main__':
    freeze_support()
    win_run()

taskWorker.py： 任务进程

import time
from multiprocessing.managers import BaseManager

class QueueManager(BaseManager):
    pass

QueueManager.register('get_task_queue')
QueueManager.register('get_result_queue')

server_addr = '127.0.0.1'
print('Connect to server %s..' % server_addr)
m = QueueManager(address=(server_addr,8001),authkey='qiye')
m.connect()
task = m.get_task_queue()
result = m.get_result_queue()

while(not task.empty()):
    image_url = task.get(True,timeout=5)
    print('run task download %s ....' % image_url)
    time.sleep(1)
    result.put('%s----->success' % image_url)

print ('work exit.')

先执行服务进程, 任务被放进 task_queue：

put task ImageUrl_0....
put task ImageUrl_1....
put task ImageUrl_2....
put task ImageUrl_3....
put task ImageUrl_4....
put task ImageUrl_5....
put task ImageUrl_6....
put task ImageUrl_7....
put task ImageUrl_8....
put task ImageUrl_9....
try get result....

服务进程还在执行时，运行任务进程，

Connect to server 127.0.0.1..
run task download ImageUrl_0 ....
run task download ImageUrl_1 ....
run task download ImageUrl_2 ....
run task download ImageUrl_3 ....
run task download ImageUrl_4 ....
run task download ImageUrl_5 ....
run task download ImageUrl_6 ....
run task download ImageUrl_7 ....
run task download ImageUrl_8 ....
run task download ImageUrl_9 ....
work exit.

任务进程结束后，可以看到数据被写入result_queue：

 

result is ImageUrl_0----->success
result is ImageUrl_1----->success
result is ImageUrl_2----->success
result is ImageUrl_3----->success
result is ImageUrl_4----->success
result is ImageUrl_5----->success
result is ImageUrl_6----->success
result is ImageUrl_7----->success
result is ImageUrl_8----->success
result is ImageUrl_9----->success