python爬虫之异步IO

在web请求非常多的情况下，异步IO往往效果和速率惊人。

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import requests,time
start_time=time.time()
resp=requests.get(url='http://httpbin.org/headers')
print(resp.text)
print('用时：%s秒'%(time.time()-start_time))

运行结果：

然后用aiohttp和asyncio进行异步IO试一下：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import asyncio, aiohttp, time
start_time=time.time()
async def request(url):
	async with aiohttp.ClientSession() as session:
		async with session.get(url) as resp:
			resp_text=await resp.read()
			print(resp_text.decode('utf-8'))
loop=asyncio.get_event_loop()
tasks=[request('http://httpbin.org/headers')]
loop.run_until_complete(asyncio.wait(tasks))
print('用时：%s秒'%(time.time()-start_time))

运行结果：

虽然只有单任务，但是IO是通过交给操作系统处理的，速度也有一点的提升。那么在多任务的情况下会怎么样：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import asyncio, aiohttp, time
start_time=time.time()
response=[]
async def fn(url):
	async with aiohttp.ClientSession() as session:
		async with session.get(url=url) as resp:
			response.append(await resp.read())
loop=asyncio.get_event_loop()
tasks=[asyncio.ensure_future(fn('http://www.baidu.com')) for x in range(500)]
loop.run_until_complete(asyncio.wait(tasks))
#print(response)
print("用时%s秒" % (time.time()-start_time))

使用协程访问500次百度，运行结果：

我们再用同步的requests试一下：

60s!