【spider04】BeautifulSoup

最新推荐文章于 2024-08-18 20:33:50 发布

原创最新推荐文章于 2024-08-18 20:33:50 发布 · 166 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Python #爬虫

爬虫专栏收录该内容

6 篇文章

订阅专栏

本文介绍了BeautifulSoup这个强大的网页解析库，详细讲解了不同解析器的优缺点，如Python标准库、lxml和html5lib。此外，还阐述了基本使用方法，包括标签选择器的各种操作，如选择元素、获取属性和内容，以及CSS选择器的应用。文章总结时推荐使用lxml解析器，并给出了使用建议。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

BeautifulSoup

灵活又方便的网页解析库，处理高效，支持多种解析器。

利用它不用编写正则表达式即可方便实现网页信息的提取。

解析器：Python标准库

使用方法：BeautifulSoup(markup, “html.parser”)

优势：Python的内置标准库执行速度适中文档容错能力强

劣势：Python 2.7.3 or 3.2.2)前的版本中文档容错能力差

解析器：lxml HTML 解析器

使用方法：BeautifulSoup(markup, “lxml”)

优势：速度快文档容错能力强

劣势：需要安装C语言库

解析器：lxml XML 解析器

使用方法：BeautifulSoup(markup, [“lxml”, “xml”]) BeautifulSoup(markup, “xml”)

优势：速度快唯一支持XML的解析器

劣势：需要安装C语言库

解析器：html5lib

使用方法：BeautifulSoup(markup, “html5lib”)

优势：最好的容错性以浏览器的方式解析文档生成HTML5格式的文档

劣势：速度慢不依赖外部拓展

基本使用

html = '''
<html><head><title>The Domouse's story</title></head>
<body>
<p class="title"name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were little sisters;and their names were
<a href="http://example.com/elsie"class="sister"id="link1"><!--Elsie--></a>
<a hred="http://example.com/lacle"class="sister"id="link2">Lacle</a>and
<a hred="http://example.com/tilie"class="sister"id="link3">Tillie</a>
and they lived at bottom of a well.</p>
<p class="story">...</p>
'''

from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.prettify())
print(soup.title.string)

<html>
 <head>
  <title>
   The Domouse's story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were little sisters;and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!--Elsie-->
   </a>
   <a class="sister" hred="http://example.com/lacle" id="link2">
    Lacle
   </a>
   and
   <a class="sister" hred="http://example.com/tilie" id="link3">
    Tillie
   </a>
   and they lived at bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
The Domouse's story

标签选择器

选择元素

html = '''
<html><head><title>The Domouse's story</title></head>
<body>
<p class="title"name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were little sisters;and their names were
<a href="http://example.com/elsie"class="sister"id="link1"><!--Elsie--></a>
<a hred="http://example.com/lacle"class="sister"id="link2">Lacle</a>and
<a hred="http://example.com/tilie"class="sister"id="link3">Tillie</a>
and they lived at bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.title)
print(type(soup.title))
print(soup.head)
print(soup.p)
# 标签选择器返回选择的第一个内容

<title>The Domouse's story</title>
<class 'bs4.element.Tag'>
<head><title>The Domouse's story</title></head>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

获取名称

html = '''
<html><head><title>The Domouse's story</title></head>
<body>
<p class="title"name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were little sisters;and their names were
<a href="http://example.com/elsie"class="sister"id="link1"><!--Elsie--></a>
<a hred="http://example.com/lacle"class="sister"id="link2">Lacle</a>and
<a hred="http://example.com/tilie"class="sister"id="link3">Tillie</a>
and they lived at bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.title.name)#最外层标签的名称

title

获取属性

html = '''
<html><head><title>The Domouse's story</title></head>
<body>
<p class="title"name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were little sisters;and their names were
<a href="http://example.com/elsie"class="sister"id="link1"><!--Elsie--></a>
<a hred="http://example.com/lacle"class="sister"id="link2">Lacle</a>and
<a hred="http://example.com/tilie"class="sister"id="link3">Tillie</a>
and they lived at bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.p.attrs['name'])
print(soup.p['name'])

dromouse
dromouse

获取内容

html = '''
<html><head><title>The Domouse's story</title></head>
<body>
<p class="title"name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were little sisters;and their names were
<a href="http://example.com/elsie"class="sister"id="link1"><!--Elsie--></a>
<a hred="http://example.com/lacle"class="sister"id="link2">Lacle</a>and
<a hred="http://example.com/tilie"class="sister"id="link3">Tillie</a>
and they lived at bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.p.string)

The Dormouse's story

嵌套选择

html = '''
<html><head><title>The Domouse's story</title></head>
<body>
<p class="title"name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were little sisters;and their names were
<a href="http://example.com/elsie"class="sister"id="link1"><!--Elsie--></a>
<a hred="http://example.com/lacle"class="sister"id="link2">Lacle</a>and
<a hred="http://example.com/tilie"class="sister"id="link3">Tillie</a>
and they lived at bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.head.title.string)

The Domouse's story

子节点和子孙节点

html = '''
<html>
   <head>
       <title>The Domouse's story</title>
   </head>
   <body>
   <p class="story">
       Once upon a time there were little sisters;and their names were
       <a href="http://example.com/elsie" class="sister"id="link1">
       <span>Elsle</span>
       </a>
       <a hred="http://example.com/lacle"class="sister" id="link2">Lacle</a>
       and
       <a hred="http://example.com/tilie"class="sister" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
       <p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.p.contents)

['\n       Once upon a time there were little sisters;and their names were\n       ', <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>, '\n', <a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>, '\n       and\n       ', <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>, '\n       and they lived at bottom of a well.\n       ']

html = '''
<html>
   <head>
       <title>The Domouse's story</title>
   </head>
   <body>
   <p class="story">
       Once upon a time there were little sisters;and their names were
       <a href="http://example.com/elsie" class="sister"id="link1">
       <span>Elsle</span>
       </a>
       <a hred="http://example.com/lacle"class="sister" id="link2">Lacle</a>
       and
       <a hred="http://example.com/tilie"class="sister" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
       <p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.p.children)# 返回迭代器 object
for i,child in enumerate(soup.p.children):
    print(i,child)

<list_iterator object at 0x000001165D636BE0>
0 
       Once upon a time there were little sisters;and their names were
       
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
2 

3 <a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
4 
       and
       
5 <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
6 
       and they lived at bottom of a well.

html = '''
<html>
   <head>
       <title>The Domouse's story</title>
   </head>
   <body>
   <p class="story">
       Once upon a time there were little sisters;and their names were
       <a href="http://example.com/elsie" class="sister"id="link1">
       <span>Elsle</span>
       </a>
       <a hred="http://example.com/lacle"class="sister" id="link2">Lacle</a>
       and
       <a hred="http://example.com/tilie"class="sister" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
       <p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.p.descendants)
for i,child in enumerate(soup.p.descendants):
    print(i,child)

<generator object descendants at 0x000001165D615620>
0 
       Once upon a time there were little sisters;and their names were
       
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
2 

3 <span>Elsle</span>
4 Elsle
5 

6 

7 <a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
8 Lacle
9 
       and
       
10 <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
11 Tillie
12 
       and they lived at bottom of a well.

父节点和祖先节点

html = '''
<html>
   <head>
       <title>The Domouse's story</title>
   </head>
   <body>
   <p class="story">
       Once upon a time there were little sisters;and their names were
       <a href="http://example.com/elsie" class="sister"id="link1">
       <span>Elsle</span>
       </a>
       <a hred="http://example.com/lacle"class="sister" id="link2">Lacle</a>
       and
       <a hred="http://example.com/tilie"class="sister" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
       <p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.a.parent)

<p class="story">
       Once upon a time there were little sisters;and their names were
       <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
<a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
       and
       <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>

html = '''
<html>
   <head>
       <title>The Domouse's story</title>
   </head>
   <body>
   <p class="story">
       Once upon a time there were little sisters;and their names were
       <a href="http://example.com/elsie" class="sister"id="link1">
       <span>Elsle</span>
       </a>
       <a hred="http://example.com/lacle"class="sister" id="link2">Lacle</a>
       and
       <a hred="http://example.com/tilie"class="sister" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
       <p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(list(enumerate(soup.a.parents)))

[(0, <p class="story">
       Once upon a time there were little sisters;and their names were
       <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
<a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
       and
       <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>), (1, <body>
<p class="story">
       Once upon a time there were little sisters;and their names were
       <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
<a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
       and
       <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
<p class="story">...</p>
</body>), (2, <html>
<head>
<title>The Domouse's story</title>
</head>
<body>
<p class="story">
       Once upon a time there were little sisters;and their names were
       <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
<a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
       and
       <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
<p class="story">...</p>
</body></html>), (3, <html>
<head>
<title>The Domouse's story</title>
</head>
<body>
<p class="story">
       Once upon a time there were little sisters;and their names were
       <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsle</span>
</a>
<a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>
       and
       <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
<p class="story">...</p>
</body></html>)]

兄弟节点

html = '''
<html>
   <head>
       <title>The Domouse's story</title>
   </head>
   <body>
   <p class="story">
       Once upon a time there were little sisters;and their names were
       <a href="http://example.com/elsie" class="sister"id="link1">
       <span>Elsle</span>
       </a>
       <a hred="http://example.com/lacle"class="sister" id="link2">Lacle</a>
       and
       <a hred="http://example.com/tilie"class="sister" id="link3">Tillie</a>
       and they lived at bottom of a well.
       </p>
       <p class="story">...</p>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(list(enumerate(soup.a.previous_siblings)))
print(list(enumerate(soup.a.next_siblings)))

[(0, '\n       Once upon a time there were little sisters;and their names were\n       ')]
[(0, '\n'), (1, <a class="sister" hred="http://example.com/lacle" id="link2">Lacle</a>), (2, '\n       and\n       '), (3, <a class="sister" hred="http://example.com/tilie" id="link3">Tillie</a>), (4, '\n       and they lived at bottom of a well.\n       ')]

标准选择器

find_all(name,attrs,recursive,text,**kwargs)

可根据标签名、属性、内容寻找文档

name

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.find_all('ul'))
print(type(soup.find_all('ul')[0]))

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
<class 'bs4.element.Tag'>

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for ul in soup.find_all('ul'):
    print(ul.find_all('li'))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

attrs

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.find_all(attrs={'id':'list-1'}))
print(soup.find_all(attrs={'name':'elements'}))

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<div class="panel-heading" name="elements">
<h4>Hello</h4>
</div>]

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.find_all(id='list-1'))
print(soup.find_all(class_='element'))#class是python关键字，须加下划线

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

text

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.find_all(text='Foo'))#返回内容，适合内容匹配，不适合元素查找

['Foo', 'Foo']

find(name,attrs,recursive,text,**kwargs)

find返回单个元素,find_all返回所有元素

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
soup = BeautifulSoup(html,'lxml')
print(soup.find('ul'))
print(type(soup.find('ul')))
print(soup.find('page'))

<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<class 'bs4.element.Tag'>
None

find_parents() find_parent()

find_parents()返回所有祖先节点，find_parents返回直接父节点

find_next_siblings() find_next_sibling()

find_next_siblings()返回后面所有兄弟节点，find__next_sibling()返回后面第一个兄弟节点。

find_previous_siblings() find_previous_sibling()

find_previous_siblings()返回前面所有兄弟节点，find__previous_sibling()返回前面第一个兄弟节点。

find_all_next() find_next()

find_all_next()返回节点后所有符合条件的节点，find_next()返回第一个符合条件的节点。

find_all_previous() find_previous()

find_all_previous()返回节点后所有符合条件的节点，find_previous()返回第一个符合条件的节点。

CSS选择器

通过select()直接传入CSS选择器即可完成选择

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.select('.panel .panel-body'))#class前加.
print(soup.select('ul li'))
print(soup.select('#list-2 .element'))#id前加#
print(type(soup.select('ul')[0]))

[<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
<class 'bs4.element.Tag'>

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for ul in soup.select('ul'):
    print(ul.select('li'))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

获取属性

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for ul in soup.select('ul'):
    print(ul['id'])
    print(ul.attrs['id'])

list-1
list-1
list-2
list-2

获取内容

html = '''
<div class="panel">
   <div class="panel-heading"name="elements">
       <h4>Hello</h4>
   </div>
   <div class="panel-body">
       <ul class="list"Id="list-1">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
           <li class="element">Jay</li>
       </ul>
       <ul class="list list-small"Id="list-2">
           <li class="element">Foo</li>
           <li class="element">Bar</li>
       </ul>
   </div>
<div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for li in soup.select('li'):
    print(li.get_text())