asyncio 异步爬虫(转)

最新推荐文章于 2025-06-13 12:32:33 发布

景霄之上

最新推荐文章于 2025-06-13 12:32:33 发布

阅读量579

点赞数

CC 4.0 BY-SA版权

分类专栏： python 文章标签：爬虫 asyncio 异步爬虫

本文链接：https://blog.youkuaiyun.com/weixin_44285988/article/details/93767632

python 专栏收录该内容

43 篇文章

订阅专栏

本文分享了一篇关于使用asyncio进行异步爬虫的代码实例，旨在方便个人日后查阅。通过asyncio库，可以提高爬虫的效率，实现并发请求，提升数据抓取速度。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

转载过来的代码,方便自己查看,勿怪

# 异步方式爬取当当畅销书的图书信息
import aiohttp
import asyncio
import time
import pandas as pd
from bs4 import BeautifulSoup

table = []

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text(encoding='gb18030')

async def parser(html):
    soup = BeautifulSoup(html,'lxml')
    book_list = soup.find('ul',class_="bang_list clearfix bang_list_mode")('li')
    for book in book_list:
        info = book.find_all('div')
        rank = info[0].text[0:-1]
        name = info[2].text
        comments = info[3].text.split('条')[0]
        author = info[4].text
        date_and_publisher = info[5].text.split()
        publisher = date_and_publisher[1] if len(date_and_publisher) >= 2 else ''

        table.append([rank,name,comments,author,publisher])

async def download(url):
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, url)
        await parser(html)

urls = ['http://bang.dangdang.com/books/bestsellers/01.00.00.00.00.00-recent7-0-0-1-%d'%i for i in range(1,26)]
print('#' * 50)
t1 = time.time()

# 利用asyncio模块进行异步IO处理
loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(download(url)) for url in urls]
tasks = asyncio.gather(*tasks)
loop.run_until_complete(tasks)

df = pd.DataFrame(table, columns=['rank','name','comments','author','publisher'])

df.to_excel('E:\\dangdang.xlsx', index=False)
t2 = time.time()
print('使用aiohttp，总共耗时：%s' % (t2 - t1))
print('#' * 50)