爬虫技术对于进行数据处理的相关研究者来说是非常重要的。要进行数据处理，首先我们必须要获取大量的数据，爬虫技术正好可以解决这个问题。对于计算机视觉方面的研究者来说，爬虫技术可以帮助我们快速构建数据集。
1.第一爬虫脚本如下：
from urllib2 import urlopen
html = urlopen("http://pythonscraping.com/pages/page1.html")
print(html.read())python scrapy.py
结果：
<html>
<head>
<title>A Useful Page</title>
</head>
<body>
<h1>An Interesting Title</h1>
<div>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</div>
</body>
</html>2.增加beatifulsoup
from urllib2 import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.pythonscraping.com/pages/page1.html")
bsObj = BeautifulSoup(html.read())
print(bsObj.h1)结果：
<h1>An Interesting Title</h1>考虑到url找不到等错误出现，修改了一下代码
from urllib2 import urlopen
from bs4 import BeautifulSoup
def getTitle(url):
try:
html=urlopen(url)
except HTTPError as e:
return None
try:
bsObj=BeautifulSoup(html.read(),'lxml')
title=bsObj.body.h1
except AttributeError as e:
return None
else:
return title
title=getTitle("http://pythonscraping.com/pages/page1.html")
if title==None:
print("Title could not be found")
else:
print(title)结果
<h1>An Interesting Title</h1>
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
