精通scrapy网络爬虫——第三章 response.xpath()and css()基础语法

最新推荐文章于 2024-04-08 16:44:58 发布

原创

最新推荐文章于 2024-04-08 16:44:58 发布 · 3.5k 阅读

28 ·

CC 4.0 BY-SA版权

本文深入讲解Scrapy框架中响应对象response的xpath()和css()基础用法，通过实例演示如何使用这两个函数进行HTML文档解析，包括绝对路径、选择子孙节点、属性选取等关键操作。

首先创建一个用于演示的HTML文档，并构造一个HtmlResponse对象

from scrapy.selector import Selector
from scrapy.http import HtmlResponse

body = '''
<html>
    <head>
        <base href='http://example.com/' />
        <title>Example website</title>
    </head>
    <body>
        <div id='image'>
            <a href='image1.html'>Name: Image 1 <br/><img src="image1.jpg"></a>
            <a href='image2.html'>Name: Image 2 <br/><img src="image1.jpg"></a>
            <a href='image3.html'>Name: Image 3 <br/><img src="image1.jpg"></a>
            <a href='image4.html'>Name: Image 4 <br/><img src="image1.jpg"></a>
            <a href='image5.html'>Name: Image 5 <br/><img src="image1.jpg"></a>
        </div>
    </body>
</html>
'''
response=HtmlResponse(url='http://example.com',body=body,encoding='utf-8')

/:描述一个从跟开始的绝对路径

#/:描述一个从跟开始的绝对路径
print(response.xpath('/html'))
print(response.xpath('/html/head'))
#[<Selector xpath='/html' data='<html>\n    <head>\n        <base href=...'>]
#[<Selector xpath='/html/head' data='<head>\n        <base href="http://exa...'>]

E1/E2:选中E1子节点的所有E2

#E1/E2:选中E1子节点的所有E2
print(response.xpath('/html/body/div/a'))
#[<Selector xpath='/html/body/div/a' data='<a href="image1.html">Name: Image 1 <...'>,
# <Selector xpath='/html/body/div/a' data='<a href="image2.html">Name: Image 2 <...'>,
# <Selector xpath='/html/body/div/a' data='<a href="image3.html">Name: Image 3 <...'>,
# <Selector xpath='/html/body/div/a' data='<a href="image4.html">Name: Image 4 <...'>,
# <Selector xpath='/html/body/div/a' data='<a href="image5.html">Name: Image 5 <...'>]

//E: 选中文档中的所有E，无论在什么位置

#//E: 选中文档中的所有E，无论在什么位置
print(response.xpath('//a'))
#[<Selector xpath='//a' data='<a href="image1.html">Name: Image 1 <...'>,
# <Selector xpath='//a' data='<a href="image2.html">Name: Image 2 <...'>,
# <Selector xpath='//a' data='<a href="image3.html">Name: Image 3 <...'>,
# <Selector xpath='//a' data='<a href="image4.html">Name: Image 4 <...'>,
# <Selector xpath='//a' data='<a href="image5.html">Name: Image 5 <...'>]

E1//E2:选中E1后代节点中的所有E2，无论在后代的什么位置，例如E1/E4/E5/E2 等价于E1//E2

#E1//E2:选中E1后代节点中的所有E2，无论在后代的什么位置，例如E1/E4/E5/E2 等价于E1//E2
print(response.xpath('/html/body//img'))
#[<Selector xpath='/html/body//img' data='<img src="image1.jpg">'>,
# <Selector xpath='/html/body//img' da