python爬虫入门豆瓣top250的抓取

最新推荐文章于 2020-08-24 11:09:27 发布

生活的涩苦

最新推荐文章于 2020-08-24 11:09:27 发布

阅读量187

点赞数

文章标签： python爬虫

本文链接：https://blog.youkuaiyun.com/qq_43606197/article/details/84222175

版权

本博客介绍了一个Python脚本，使用requests和BeautifulSoup库从豆瓣网站爬取电影Top250榜单，并将结果存储到本地文件中。通过发送HTTP请求获取网页内容，解析HTML结构提取电影名称。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import requests
from bs4 import BeautifulSoup

def get_page():
	headers = {
		'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 '
					  '(KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36'
	}
	movies = []
	for i in range(10):
		url = 'https://movie.douban.com/top250?start=' + str(i * 25)
		response = requests.get(url, headers=headers)
		html = response.text
		soup = BeautifulSoup(html,'lxml')
		div_list = soup.find_all('div', class_='hd')
		for item in div_list:
			movie = item.a.span.text.strip()
			movies.append(movie)

	print(movies)
	length = len(movies)
	for j in range(0, length):
		fileHandle = open("movies.txt", 'a', encoding='utf-8')
		fileHandle.write(str(j+1)+ ':' + movies[j])
		fileHandle.write("\n")
get_page()