python爬虫爬取豆瓣TOP250用csv文件

踏过星空

已于 2024-06-06 18:30:27 修改

阅读量1.5k

点赞数 16

CC 4.0 BY-SA版权

文章标签： python 爬虫开发语言

于 2024-06-06 18:25:37 首次发布

本文链接：https://blog.youkuaiyun.com/life12345678910/article/details/139507311

一、爬取数据并用csv文件保存

import numpy as np
import requests
from lxml import etree
from time import sleep
import xlwt
import csv
url='https://movie.douban.com/top250'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
}
titles_cn = []
titles_en=[]
links = []
director=[]
actors=[]
years=[]
nations=[]
types=[]
scores=[]
rating_nums=[]

fp = open('./douban_top250.csv','w',encoding='utf-8')
writer = csv.writer(fp)
writer.writerow(
    ['电影中文名','电影英文名','电影详情页链接','导演','演员','上映年份','国际','类型','评分','评分人数']
)
for i in range(0,226,25):
    url = f'https://movie.douban.com/top250?start={i}&filter='
    data={
        'start':i,
        'filter':' ',
    }
    response = requests.get(url, headers=headers, data=data)
    sleep(1)
    #print(response, status_code)
    #print(response, encoding)
    #print(response.text)
    html = response.text
    data = etree.HTML(html)
    li_list=data.xpath('//*