网络编程 - 协议遇到IO自动切换-优快云博客

本文探讨了Python网络编程中使用gevent模块实现IO自动切换的机制，通过示例展示了如何利用gevent进行网页并发爬取，显著提高效率。

一、协议遇到IO自动切换

python网络编程，遇到IO自动切换，通过模块gevent来实现；

import gevent,time

def g1():
    print ("g1 is start!")
    gevent.sleep(3)
    print ("g1 is end!")
def g2():
    print ("g2 is start!")
    gevent.sleep(2)
    print ("g2 is end!")
def g3():
    print ("g3 is start!")
    gevent.sleep(1)
    print ("g3 is end!")

gevent.joinall([gevent.spawn(g1),
                gevent.spawn(g2),
                gevent.spawn(g3)])
二、协议gevent并发爬网页

import gevent,time
from urllib import request
from gevent import monkey
monkey.patch_all()#把当前io的所有操作做上标记（如果未导入该语句，gevent无法确认urllib是io操作）；

def f(url):
    print ("GET %s" % url)
    res =request.urlopen(url)
    data = res.read()
#     f= open("a.html","wb")
#     f.write(data)
#     f.close()
    print ("%d bytes received from %s" %(len(data),url))
urls = ["https://www.python.org/",
        "https://www.yahoo.com/",
        "https://www.github.com/"]
time_start = time.time()
for url in urls:
    f(url)
total_time = time.time()-time_start
print (total_time)
time_start1 = time.time()
gevent.joinall([
    gevent.spawn(f,"https://www.python.org/"),
    gevent.spawn(f,"https://www.yahoo.com/"),
    gevent.spawn(f,"https://www.github.com/")
])
async_time = time.time() - time_start1
print (async_time)

运行结果：

GET https://www.python.org/
48990 bytes received from https://www.python.org/
GET https://www.yahoo.com/
508828 bytes received from https://www.yahoo.com/
GET https://www.github.com/
79979 bytes received from https://www.github.com/
5.865999937057495
GET https://www.python.org/
GET https://www.yahoo.com/
GET https://www.github.com/
508974 bytes received from https://www.yahoo.com/
48990 bytes received from https://www.python.org/
79979 bytes received from https://www.github.com/
1.371999979019165

Process finished with exit code 0

从结果中，可以看出，for循环，默认为串行操作；gevent.joinall操作，默认遇到IO操作时，自动切换成并行；

转载于:https://www.cnblogs.com/wulafuer/p/10281910.html