Python 3 爬虫 豆瓣电影top250(requests,BeautifulSoup解析)

1.使用requests,和BeautifulSoup解析

import requests
from bs4 import BeautifulSoup


HEADERS = {
    "Referer": "https://movie.douban.com/top250",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/65.0.3325.181 Safari/537.36"
}
html = requests.get("https://movie.douban.com/top250?start=0&filter=")
bsobj = BeautifulSoup(html.content, 'lxml')
with open(r"C:\Users\jone\Desktop\douban.txt", 'w') as fs:
    fs.write(str(bsobj.contents).encode('gb18030').decode('gbk', 'ignore'))

print("done")

2.问题解析：

1.模拟浏览器登陆，添加头信息

2.BeautifulSoup无法解析Response响应，创建BeautifulSoup时，应该解析html.content

3.将网页源码写入txt文件时候，将网页内容转化为字符串后，先encode成gb18030,再解码为gbk,选择忽略不能识别的字符。