爬虫实战——古诗文网古诗爬取

最新推荐文章于 2024-10-29 15:18:58 发布

原创最新推荐文章于 2024-10-29 15:18:58 发布 · 1.4k 阅读

6 ·

CC 4.0 BY-SA版权

文章标签：

#python

python爬虫学习专栏收录该内容

3 篇文章

订阅专栏

本文介绍了一种使用正则表达式和requests库从古诗文网站爬取诗词信息的方法，包括标题、朝代、作者及内容。通过解析网页源代码，实现了对1到10页诗词数据的有效抓取。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import re
import requests

# url = 'https://www.gushiwen.org/default_1.aspx'


def parse_page(url):
    headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'
    }
    response = requests.get(url, headers)
    text = response.text
    # print(text)
    titles = re.findall(r'<div\sclass="cont">.*?<b>(.*?)</b>', text, re.DOTALL)
    # print(titles)
    dynasties = re.findall(r'<p\sclass="source".*?<a.*?>(.*?)</a>', text, re.DOTALL)
    # print(dynasties)
    authors = re.findall(r'<p\sclass="source".*?</a>.*?<a.*?>(.*?)</a>', text, re.DOTALL)
    # print(authors)
    contents = re.findall(r'<div\sclass="contson".*?>(.*?)</div>', text, re.DOTALL)
    # print(contents)
    poems = []
    for content in contents:
        # print(content)
        x = re.sub(r'<.*?>', "", content)
        # print(x)
        poems.append(x.strip())
        # print(poems)
        shis = []
    for value in zip(titles, dynasties, authors, poems):
        title, dynasty, author, poem = value
        shi = {
            'title': title,
            'dyna': dynasty,
            'author': author,
            'poem': poem
        }
        shis.append(shi)
        for shi in shis:
            print(shi)
            print("="*30)


def main():
    base_url = 'https://www.gushiwen.org/default_{}.aspx'

    for x in range(1, 11):	# 利用正则表达式爬取1-10页的内容
        url = base_url.format(x)
        parse_page(url)


if __name__ == '__main__':
    main()

正则表达式爬取古诗文网的诗词