csdn博客爬虫更新

最新推荐文章于 2020-05-12 16:39:52 发布

原创最新推荐文章于 2020-05-12 16:39:52 发布 · 427 阅读

0 ·

CC 4.0 BY-SA版权

几天没上csdn博客，不知道为什么给我csdn首页改了，不是以前的网页布局了，所以之前写的csdn博客爬虫也就宣告失效，所以今天修改了下之前写的xpath爬虫，正则爬虫就没改了，改的有点麻烦

# -*- coding:gbk -*-
import sys
import requests
import re
from lxml import etree
from lxml import html as ht

def download(url):
    headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"}
    html=requests.get(url,headers=headers).text
    return html

html2=download("http://blog.youkuaiyun.com/Joliph")
selector2=etree.HTML(html2)
pagelist=selector2.xpath('//*[@id="papelist"]/a[last()-2]/text()')[0]
#这里有有个潜在的问题，在我博客写到5页以上时出现...后无法判断页数
pagelist=int(pagelist)
for page in range(1,pagelist+1):
    url="http://blog.youkuaiyun.com/Joliph/article/list/"+str(page)
    html=download(url)
    selector=etree.HTML(html)
    titlelist=selector.xpath('//*[@class="link_title"]/a/text()')
    datelist=selector.xpath('//*[@class="article_manage"]/span[1]/text()')
    #/text()!!!!!!!!!!!!!!!!!!!!!!!
    number=len(titlelist)
    for i in range(1,number+1):
        tree=ht.fromstring(titlelist[i-1])
        strcom=tree.xpath('string(.)')
        print(datelist[i-1]+"----"+strcom)