w2 Advanced HTML Parsing( including Regexpression )

本文详细介绍了使用Python的BeautifulSoup库进行网页数据抓取的方法,包括解析HTML文档、查找特定元素、提取图片链接及处理复杂属性等高级技巧,是网络爬虫开发者的实用指南。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

################################################################

from urllib.request import urlopen
from bs4 import BeautifulSoup

html=urlopen('http://www.pythonscraping.com/pages/page3.html')
bs=BeautifulSoup(html, 'html.parser')

#find_all(tag, attributes, recursive, text, limit, keywords)

nameList = bs.findAll('span', {'class':'green'})

for name in nameList:

      print(name.get_text())

nameList = bs.find_all(text='the prince')########
print(len(nameList))

for child in bs.find('table', {'id':'giftList'}).children:
    print(child)

for sibling in bs.find('table', {'id':'giftList'}).tr.next_siblings:
    print(sibling)
                     #<img src="../img/gifts/img1.jpg">
print(bs.find('img', {'src':'../img/gifts/img1.jpg'}
             ).parent.previous_sibling.get_text()
            )#   $15.00

#<td>$15.00</td>
#<td><img src="../img/gifts/img1.jpg"></td></tr>
images = bs.find_all('img', {'src':re.compile('\.\.\/img\/gifts\/img.*\.jpg')})
#images = bs.find_all('img', {'src':re.compile('\.\.\/img\/gifts/img.*\.jpg')})

for image in images:
    print(image['src'])

# the following retrieves all tags that have exactly two attributes:

print(
        bs.find_all(lambda tag: len(tag.attrs)==2)
)

That is,
it will find tags with two attributes, such as the following:
<div class="body" id="content"></div>
<span style="color:red" class="title"></span>

################################################################

bs.find_all(lambda tag: tag.get_text() ==
'Or maybe he\'s only resting?')

This can also be accomplished without a lambda function:
bs.find_all('', text='Or maybe he\'s only resting?')

################################################################

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen('http://www.pythonscraping.com/pages/warandpeace.html')
bs = BeautifulSoup(html.read(), 'html.parser')

#find_all(tag, attributes, recursive, text, limit, keywords)
nameList = bs.findAll('span', {'class':'green'})
for name in nameList:
    print(name.get_text())

nameList = bs.find_all(text='the prince') ############
print(len(nameList))

Result:

http://www.pythonscraping.com/pages/warandpeace.html

...

################################################################

from urllib.request import urlopen
from bs4 import BeautifulSoup

html=urlopen('http://www.pythonscraping.com/pages/page3.html')
bs=BeautifulSoup(html, 'html.parser')
                                                #.descendants
for child in bs.find('table', {'id':'giftList'}).children:
    print(child)

Result:

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen('http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, 'html.parser')

for sibling in bs.find('table', {'id':'giftList'}).tr.next_siblings:
    print(sibling)

Result:

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen('http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, 'html.parser')

                     #<img src="../img/gifts/img1.jpg">
print(bs.find('img', {'src':'../img/gifts/img1.jpg'}
             ).parent.previous_sibling.get_text()
            )#   $15.00

#<td>$15.00</td>
#<td><img src="../img/gifts/img1.jpg"></td></tr>

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

html = urlopen('http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, 'html.parser')

images = bs.find_all('img', {'src':re.compile('\.\.\/img\/gifts\/img.*\.jpg')})

for image in images:
    print(image['src'])

print(
        #bs.find_all(lambda tag: len(tag.attrs)==2)
        bs.find_all(lambda tag: tag.get_text() == 'Or maybe he\'s only resting?')
)

评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

LIQING LIN

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值