BeatifulSoup库入门

最新推荐文章于 2023-07-17 22:44:13 发布

the_power

最新推荐文章于 2023-07-17 22:44:13 发布

阅读量559

点赞数 1

CC 4.0 BY-SA版权

文章标签： BeautifulSoup python html解析

本文链接：https://blog.youkuaiyun.com/the_power/article/details/86658023

本文介绍了BeautifulSoup库的入门知识，包括安装与导入、解析器选择、BeautifulSoup类的基本元素如Tag标签、NavigableString、Comment，以及如何进行HTML内容的遍历，包括下行、上行和平行遍历。此外，还提到了bs4库的编码处理。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

BeautifulSoup库入门

Beautiful Soup库是解析、遍历、维护“标签树”的功能库

BeautifulSoup库的安装与导入

win平台，“以管理员身份运行”cmd,执行

pip install beautifulsoup4

导入方式

from bs4 import BeautifulSoup

BeautifulSoup库解析器

bs4的HTML解析器

安装bs4库

BeautifulSoup(mk,‘html.parser’)

lxml的HTML解析器

pip install lxml

BeautifulSoup(mk,‘lxml’)

lxml的XML解析器

pip install lxml

BeautifulSoup(mk,‘xml’)

html5lib解析器

pip install html5lib

BeautifulSoup(mk,‘html5lib’)

BeautifulSoup类的基本元素

Tag标签

基本元素	说明
Tag	标签，最基本的信息组织单元，分别用<>和</>标明开通和结尾

from bs4 import BeautifulSoup
soup = BeautifulSoup('<a><span>hi</span></a>','html.parser')
soup.a

<a><span>hi</span></a>

注意：当文档中存在多个相同的对应内容时，返回第一个

Tag的name（名字）

基本元素	说明
Name	标签的名字,`<a>..</a>`的名字是’p’

soup = BeautifulSoup('<a><span>hi</span></a>','html.parser')
soup.a.name

'a'

Tag的attrs（属性）

基本元素	说明
Attributes	标签的属性,字典形式组织，格式：`<tag>.attrs`

soup = BeautifulSoup('<a href="#">test</a>','html')
soup.a.attrs

{'href': '#'}

Tag的NavigableString

基本元素	说明
NavigableString	标签内非属性字符串,`<>..</>`字符串，格式：`<tag>.string`

soup = BeautifulSoup('<a>this is a string</a>','html')
print(soup.string)
print(soup.a.string)

this is a string
this is a string

Tag的Comment

基本元素	说明
Comment	标签内字符串注释部分，一种特殊的Comment类型

soup = BeautifulSoup('<b><!--this is a comment--></b><p>this is note a comment</p>','html')
print(soup.b.string)
print(soup.p.string)
print()
#comment是一种特殊类型
print( type(soup.b.string) )
print( type(soup.p.string) )

this is a comment
this is note a comment

<class 'bs4.element.Comment'>
<class 'bs4.element.NavigableString'>

基于bs4库的HTML内容遍历方法

BeautifulSoup类型是标签树的根节点

#初始化soup
soup = BeautifulSoup('<html><head><title>This is a python demo page</title></head><body><pclass="title"><b>The demo python introduces several python courses.</b></p><p class=“course”>Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:<a href="http://www.icourse163.org/course/BIT‐268001" class="py1" id="link1">Basic Python</a>and <a href="http://www.icourse163.org/course/BIT‐1001870001" class="py2" id="link2">Advanced Python</a>.</p></body></html> ','html')
print(soup.prettify()) #使得HTML内容更加友好的显示（格式化显示）

<html>
 <head>
  <title>
   This is a python demo page
  </title>
 </head>
 <body>
  <pclass>
   <b>
    The demo python introduces several python courses.
   </b>
   <p class="“course”">
    Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:
    <a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">
     Basic Python
    </a>
    and
    <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">
     Advanced Python
    </a>
    .
   </p>
  </pclass>
 </body>
</html>

标签树下行遍历

属性	说明
.contents	子节点的列表，将`<tag>`所有儿子节点存入列表
.children	子节点的地带类型，与 .contents类似，用于循环遍历儿子节点
.descendants	子孙节点的迭代类型，包含所有的子孙节点，用于循环遍历

print("soup.head is:\n",soup.head)
print("\n soup.body.contents is:\n",soup.body.contents)
print("\n soup.body.contents lenth:\n",len(soup.body.contents))

soup.head is:
 <head><title>This is a python demo page</title></head>

 soup.body.contents is:
 [<pclass><b>The demo python introduces several python courses.</b><p class="“course”">Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">Basic Python</a>and <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>.</p></pclass>]

 soup.body.contents lenth:
 1

#遍历儿子节点
for child in soup.body.children:
    print(child)

<pclass><b>The demo python introduces several python courses.</b><p class="“course”">Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">Basic Python</a>and <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>.</p></pclass>

#遍历子孙节点
for child in soup.body.descendants:
    print(child)

<pclass><b>The demo python introduces several python courses.</b><p class="“course”">Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">Basic Python</a>and <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>.</p></pclass>
<b>The demo python introduces several python courses.</b>
The demo python introduces several python courses.
<p class="“course”">Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">Basic Python</a>and <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>.</p>
Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">Basic Python</a>
Basic Python
and 
<a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>
Advanced Python
.

标签树的上行遍历

属性	说明
.parent	节点的父亲标签
.parents	节点先辈的迭代类型，用于循环遍历先辈节点

print("soup.title.parent is :\n",soup.title.parent)
print("soup.html.parent is :\n",soup.html.parent)
print("soup.parent is :\n",soup.parent)

soup.title.parent is :
 <head><title>This is a python demo page</title></head>
soup.html.parent is :
 <html><head><title>This is a python demo page</title></head><body><pclass><b>The demo python introduces several python courses.</b><p class="“course”">Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT‐268001" id="link1">Basic Python</a>and <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>.</p></pclass></body></html> 
soup.parent is :
 None

for parent in soup.a.parents:
    if parent is None:
        print(parent)
    else:
        print(parent.name)

p
pclass
body
html
[document]

标签树的平行遍历

属性	说明
.next_sibling	返回按照HTML文本顺序的下一个平行节点标签
.previous_sibling	返回按照HTML文本顺序的上一个平行节点标签
.next_siblings	迭代类型，返回按照HTML文本顺序的后续所有平行节点标签
.previous_siblings	迭代类型，返回按照HTML文本顺序的前续所有平行节点标签

注意：平行遍历发生在同一个父节点下的各个节点间

print("soup.a.next_sibling is :\n",soup.a.next_sibling)
print("soup.a.next_sibling.next_sibling is :\n",soup.a.next_sibling.next_sibling)
print("\n soup.a.previous_sibling :\n",soup.a.previous_sibling)
print("soup.a.previous_sibling is :\n",soup.a.previous_sibling.previous_sibling)

soup.a.next_sibling is :
 and 
soup.a.next_sibling.next_sibling is :
 <a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>

 soup.a.previous_sibling :
 Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:
soup.a.previous_sibling is :
 None

#遍历后序节点
for sibling in soup.a.next_siblings:
    print(sibling)

and 
<a class="py2" href="http://www.icourse163.org/course/BIT‐1001870001" id="link2">Advanced Python</a>
.

#遍历前序节点
for sibling in soup.a.previous_siblings:
    print(sibling)

Python is a wonderful general‐purpose programming language.You can learn Python from novice to professional by tracking the following courses:

bs4库的编码

bs4库将任何HTML的输入都变成utf-8编码
python 3.x默认支持编码是utf-8，解析无障碍