古诗文网站之网络爬虫

最新推荐文章于 2024-10-29 15:18:58 发布

原创

最新推荐文章于 2024-10-29 15:18:58 发布 · 2.5k 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#爬虫

1. 下面是古诗文网站的网络爬虫代码

# encoding:utf-8
import requests
import re
import json


def parse_page(url):
    # 1.请求网站
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36"
    }
    response = requests.get(url, headers=headers)
    text = response.text
    # 2.解析网站
    titles = re.findall(r'<div\sclass="cont">.*?<b>(.*?)</b>', text, re.DOTALL)
    # print json.dumps(titles, encoding="utf-8", ensure_ascii=False)
    times = re.findall(r'<p\sclass="source">.*?<a\s.*?>(.*?)</a>', text, re.DOTALL)
    # print json.dumps(times, encoding="utf-8", ensure_ascii=False)
    authors = re.findall(r'<p class="source">.*?<a.*?<a.*?>(.*?)</a>', text, re.DOTALL)
    poems_ret = re.findall(r'<div class="contson" id=.*?>(.*?)</div>', text, re.DOTALL)
    poems = []
    for poem in poems_ret:
        temp