Python爬虫实例（二）爬取数据后并入excel

最新推荐文章于 2025-09-07 13:57:25 发布

原创最新推荐文章于 2025-09-07 13:57:25 发布 · 1.6k 阅读

5 ·

CC 4.0 BY-SA版权

python爬虫专栏收录该内容

2 篇文章

订阅专栏

本文介绍了一种使用Python爬虫抓取豆瓣电影Top250榜单的方法，并通过pandas库将数据整理后保存为Excel文件。文章详细展示了如何设置请求头、使用requests库发起HTTP请求、利用lxml库解析HTML文档，以及如何处理可能出现的模块未找到错误。

报错：Module Not Found Error: No module named 'openpyxl'

则下载openpyxl模块，pip install openpyxl

源码：

import requests
from lxml import etree
import pandas as pd
#设置请求头
headers = {
   'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0'
}
cookies = {'cookie':'"108300"; bid=x6CWhyewqig; __utma=30149280.485109852.1574991282.1574991282.1574991282.1; __utmc=30149280; __utmz=30149280.1574991282.1.1.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; ap_v=0,6.0; gr_user_id=056ec422-999e-4cc8-82e1-4757d6547298; gr_session_id_22c937bbd8ebd703f2d8e9445f7dfd03=a3df6afd-13f6-4470-b577-a185c4c218e3; gr_cs1_a3df6afd-13f6-4470-b577-a185c4c218e3=user_id%3A0; gr_session_id_22c937bbd8ebd703f2d8e9445f7dfd03_a3df6afd-13f6-4470-b577-a185c4c218e3=true; _vwo_uuid_v2=DEE210CAD2A1C37C8754AAD0465FEFC45|3c7ca2da3525dbfd34f59bad5030fbde; __gads=ID=9438760ce15e89b2:T=1574991108:S=ALNI_MbPmQ5P86tmDdlvMJS_0so-WCfqzQ; __utmb=30149280.4.10.1574991282'}
#保存列表
film = []
#封装url请求方法
def get_film(url,headers,cookies):
    # 发送get请求
    response = requests.get(url, cookies=cookies, headers=headers).text
    # 获取解析结果,etrrr.HTML只能解析str
    tree = etree.HTML(response)
    name_list = tree.xpath('.//div[@class="hd"]/a/span[@class="title"][1]/text()')
    film.extend(name_list)

if __name__ == '__main__':
    for value in list(range(0, 250, 25)):
        # 生成每一页的地址
        url = "https://movie.douban.com/top250?start=" + str(value) + "&filter="
        get_film(url, headers, cookies)
    #使用pandas模块将数据保存到excel文件中
    data1 = pd.DataFrame({
        "电影名称":film
    })
    #将数据写入到excel表格中
    data1.to_excel(excel_writer="豆瓣电影top250.xlsx",sheet_name='电影排行',columns=['电影名称'])
    print(film)