Python 爬虫 xpath 和 bs4

Catalog

xpath

lxml模块中的xpath

html = '''<body>
<p class="a">A</p>
<p class="b">
<a href="https://blog.youkuaiyun.com/Yellow_python">B</a>
</p>
</body>'''
from lxml import etree
# 创建【Element】对象
element = etree.HTML(html, etree.HTMLParser())
# xpath 提取信息，返回列表
text = element.xpath('//body/p[1]/text()')
print(text)
href = element.xpath('//body/p/a/@href')
print(href)

打印结果

[‘A’]
[‘ https://blog.youkuaiyun.com/Yellow_python‘]

scrapy中的xpath

1. 运行终端
2. scrapy shell https://blog.youkuaiyun.com/Yellow_python
3. response.xpath(”).extract()

response.xpath(‘/html/body/header/div/div[1]/h2/a/text()’).extract_first()
‘Yellow_python的博客’

BeautifulSoup

html = '''<body>
<p class="a">A</p>
<p class="b">
<a href="https://blog.youkuaiyun.com/Yellow_python">B</a>
</p>
</body>'''
from bs4 import BeautifulSoup
# 创建 bs 对象
bs = BeautifulSoup(html, 'html.parser')
# 提取信息
first = bs.find('a')
# 查找第1个
print(first)
print(type(first))  # <class 'bs4.element.Tag'>
print(first.get_text())
# 查找全部，返回列表
all = bs.find_all('p', {'class': 'a'})
print(all)
print(type(all))  # <class 'bs4.element.ResultSet'>
# 转字符串
prettify = bs.prettify()
print(prettify)
print(type(prettify))  # <class 'str'>