利用python 爬取豆瓣即将上映的电影

最新推荐文章于 2025-02-13 10:48:27 发布

god_hastur

最新推荐文章于 2025-02-13 10:48:27 发布

阅读量530

点赞数 1

文章标签： python

本文链接：https://blog.youkuaiyun.com/god_hastur/article/details/119544217

版权

该代码示例展示了如何使用Python爬虫从豆瓣电影网站抓取即将上映电影的基本信息，如标题、时间、类型、地区和热度，并通过自定义的doubanapi模块调用API获取电影的导演、演员和简介。最后，将所有信息整合到一个字典中并保存为列表。整个过程涉及网络请求、正则表达式、BeautifulSoup解析和JSON数据处理。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

仅学习交流

Upcoming.py

import re
import requests
from doubanapi import findmovie
from bs4 import BeautifulSoup
url = "https://movie.douban.com/cinema/later/taian/" 
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}
res = requests.get(url=url, headers=headers)
res.encoding = 'utf-8'
soup = BeautifulSoup(res.text,'lxml')
movies = []
for sp in soup.find_all('div',class_= re.compile('item mod.*')):
    weber = sp.div.h3.a.get('href')
    title = sp.div.h3.a.string
    sp2 = sp.find_all('li')
    mtime = sp2[0].string
    mtype = sp2[1].string
    mlocat = sp2[2].string
    mpeople = sp2[3].string
    mid = re.sub("\D", "", weber)
    lits = findmovie(mid)
    dire = lits[0]
    actr = lits[1]
    content = lits[2]
    movie = {
        "标题":title,
        "时间":mtime,
        "类型":mtype,
        "地区":mlocat,
        "热度":mpeople,
        "导演":dire,
        "演员":actr,
        "简介":content,
        "详细信息":weber
    }
    print(movie)
    movies.append(movie)
print(movies)

doubanapi.py(调用api对电影信息进行补充)

import json
import requests

def findmovie(urlid):
    baseurl = 'https://movie.querydata.org/api?id='
    url = baseurl + urlid
    res = requests.get(url)
    res.encoding = 'utf-8'
    lis = res.text
    lis = json.loads(lis)
    print(res.text)        #测试
    content = lis['data'][0]['description']
    director = lis['director']
    actors = lis['actor']
    sumdire = ""
    sumact = ""
    for dire in director:
        sumdire += dire["data"][0]['name'] + " "
    for act in actors:
        sumact += act["data"][0]['name'] + " "
    sumer = {}
    sumer[0] = sumdire
    sumer[1] = sumact
    sumer[2] = content
    return sumer