bs4 爬虫

最新推荐文章于 2025-07-10 14:09:07 发布

lswzw

最新推荐文章于 2025-07-10 14:09:07 发布

阅读量682

点赞数

CC 4.0 BY-SA版权

分类专栏： Python 文章标签： py

本文链接：https://blog.youkuaiyun.com/lswzw/article/details/91048072

Python 专栏收录该内容

21 篇文章

订阅专栏

本文介绍了一种利用Python的BS4库进行网页爬取的方法，详细展示了如何从指定URL获取页面内容，解析HTML结构，抓取章节标题与链接，并下载章节内容到本地文件。通过设置请求头和编码方式，确保了爬虫能够顺利运行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

bs4 爬虫

import requests
from bs4 import BeautifulSoup

if __name__=='__main__':
    target="http://192.168.5.55/1.html"
    save_path = 'Z:\\text'
    index_path='https://www.wuruo.com/126/126892/'
    req=requests.get(url=target,
                     headers={
            'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36'
        }  
                     )
    req.encoding = 'gbk'
    soup = BeautifulSoup(req.text,"html.parser")
    #print(soup)
    list_tag = soup.find('div',attrs={'id':'readerlist'})
    #print(list_tag)
        
for dd_tag in list_tag.find_all('li',attrs={'a':''}):
    chapter_name = dd_tag.a.string
    #print(chapter_name)
    chapter_url=index_path+dd_tag.a.get('href')
    #print(chapter_url)
    chapter_req = requests.get(
        url=chapter_url,
        headers={
            'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36'
        }  
    )
    chapter_req.encoding = 'gbk'
    chapter_soup = BeautifulSoup(chapter_req.text, "html.parser")
    #print(chapter_soup)
    content_tag = chapter_soup.find('div',attrs={'id':'content'})
    content_name = chapter_soup.title.string
    print(content_name)
    content_text = str(content_tag.text.replace('\xa0','\n'))
    #print(content_text)
    with open('Z:\\0\\1.txt', 'a') as f:
            f.write(content_name)
            f.write(content_text)