requests库爬取豆瓣电影信息-优快云博客

本文链接：https://blog.youkuaiyun.com/qq_40298768/article/details/108548600

本文介绍如何利用Python的requests和BeautifulSoup库抓取豆瓣电影网站上的数据，详细解析了爬虫实现步骤，包括发送HTTP请求、解析HTML内容等关键环节。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

使用requests和BeautifulSoup库爬取豆瓣电影https://movie.douban.com/相关信息

import requests
from bs4 import BeautifulSoup
import random
import bs4
import functools

my_headers = [
    "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3741.400 QQBrowser/10.5.3863.400"
    "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36",
    "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14",
    "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)",
    'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11',
    'Opera/9.25 (Windows NT 5.1; U; en)',
    'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
    'Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Kubuntu)',
    'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.12) Gecko/20070731 Ubuntu/dapper-security Firefox/1.5.0.12',
    'Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/1.2.9',
    "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.7 (KHTML, like Gecko) Ubuntu/11.04 Chromium/16.0.912.77 Chrome/16.0.912.77 Safari/535.7",
    "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0 "
]
headers = {
   'User-Agent': random.choice(my_headers)}


def fans_num(url_offset):  # 获取某人的粉丝数
    try:
        resp = requests.get("https://movie.douban.com" + url_offset, headers=headers)
        resp.encoding = "utf-8"
        s = BeautifulSoup(resp.text, "html.parser")
        return str(s.find('div', id="fans", class_="mod").h2).split('\n')[1].split('（')[1].split('）')[0]
    except:
        return 0


def ave_all_movies(url_offset)