协程

最新推荐文章于 2024-12-04 21:27:46 发布

原创最新推荐文章于 2024-12-04 21:27:46 发布 · 234 阅读

0 ·

CC 4.0 BY-SA版权

爬虫专栏收录该内容

17 篇文章

订阅专栏

本文介绍如何利用Python第三方包gevent实现协程并发处理网络请求，通过示例代码展示了如何使用gevent.spawn启动任务并使用gevent.joinall等待所有任务完成，以及如何结合gevent.pool.Pool限制并发数量。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

# 协程

#第三方包gevent
from gevent import monkey;monkey.patch_all()
import gevent
import urllib2

def run_task(url):
    print('Visit -->>%s'%url)
    try:
        response = urllib2.urlopen(url)
        data = response.read()
        print('%d bytes received from %s'%(len(data),url))
    except Exception.e:
        print(e)

if __name__=='__mian__':
    urls = ['https://github.com/','https://www.python.org/','http://www.cnblogs.com/']
    greenlets = [gevent.spawn(run_task,url) for url in urls]
    gevent.joinll(greenlets)

在这里插入图片描述

from gevent import monkey
monkey.patch_all()
import urllib2
from gevent.pool import Pool
def run_task(url):
    print('Visit-->>%s'%url)
    try:
        response = urllib2.urlopen(url)
        data = response.read()
        print('%d bytes received from %s'%(len(data),url))
    except Exception.e:
        print(e)
    return ('url:%s---->>finish'%url)

if __name__=='__main__':
    pool = Pool(2)
    urls = ['https://github.com/','https://www.python.org/','http://www.cnblogs.com/']
    results = pool.map(run_task,urls)
    print(results)