python BeautifulSoup 抓取python中文开发者社区中的所有高级教程

最新推荐文章于 2024-01-09 11:44:36 发布

抉择命运

最新推荐文章于 2024-01-09 11:44:36 发布

阅读量1.3k

点赞数

CC 4.0 BY-SA版权

文章标签： python

本文链接：https://blog.youkuaiyun.com/u014245412/article/details/53410039

本文介绍如何运用Python的BeautifulSoup库高效地抓取Python中文开发者社区内的高级教程资源，通过实例代码详细解析抓取过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

话不多说直接上代码：

#coding=utf-8
from bs4 import BeautifulSoup
import urllib2

url = 'http://www.pythontab.com/html/pythonhexinbiancheng/index.html'
url_list = [url]
for i in range(2,19):
    url_list.append('http://www.pythontab.com/html/pythonhexinbiancheng/%s.html'%i)
source_list = []
for j in url_list:
    request = urllib2.urlopen(j)
    html = request.read()
    suop = BeautifulSoup(html,'lxml')
    titles = suop.select('#catlist > li > a')
    links = suop.select('#catlist > li > a')
    for title, link in zip(titles, links):
        data = {
            "title" : title.get_text(),
            "link" : link.get('href')
        }
        source_list.append(data)
    for l in source_list:
        request = urllib2.urlopen(l['link'])
        html = request.read()
        suop = BeautifulSoup(html,'lxml')
        text_p = suop.select('#Article > div.content > p')
        text = []
        print(text_p)
        for t in text_p:
            text.append(t.get_text().encode('utf-8'))
        title_text  = l['title']
        title_text = title_text.replace('*','').replace('/','or').replace('"',' ').replace('?','wenhao').replace(':',' ')

        with open('%s.txt'%title_text, 'wb') as f:
            for a in text:
                f.write(a)