requests库爬虫
1 安装模块
首先需要安装一下request的爬虫库
1.1 使用pip安装requests
pip install requests
2 简单爬取网页的源代码
2.1 引入模块
#引入模块
import requests
2.2 获取网页的状态（404、500、200等）
#爬取网页源代码
url = "https://www.youkuaiyun.com/"#需要爬取的网页
r = requests.get(url)#获取状态
print("状态：\n"+str(r)+"\n\n")#“r”不是一个字符串，需要加上“str()”
运行结果：
状态：
<Response [200]>
2.3 获取网页源码
print("网页代码\n"+r.text.encode(r.encoding).decode()+"\n\n")
获取网页代码使用“r.text”，使用“.encode(r.encoding).decode()”从二进制转字符串
运行结果：
网页代码
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no">
<meta name="apple-mobile-web-app-status-bar-style" content="black">
<meta name="referrer"content="always">
<meta name="msvalidate.01" content="3189512127C34C46BC74BED5852D45E4" />
<title>优快云 - 专业开发者社区</title>
<meta data-n-head="true" data-hid="description" name="description" content="优快云是全球知名中文IT技术交流平台,创建于1999年,包含原创博客、精品问答、职业培训、技术论坛、资源下载等产品服务,提供原创、优质、完整内容的专业IT技术开发社区.">
<script src='//g.csdnimg.cn/tingyun/1.8.3/www.js' type='text/javascript'></script>
<link ref="canonical" href="https://www.youkuaiyun.com/">
<link href="//csdnimg.cn/public/favicon.ico" rel="SHORTCUT ICON">
<link rel="stylesheet" href="//csdnimg.cn/public/common/toolbar/content_toolbar_css/content_toolbar.css">
<link rel="stylesheet" href="//csdnimg.cn/public/common/libs/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="//csdnimg.cn/public/static/css/avatar.css">
<link href="//g.csdnimg.cn/nav-second/1.0.2/css/nav-second.css" type="text/css" rel="stylesheet">
<script src="//csdnimg.cn/public/common/libs/jquery/jquery-1.9.1.min.js" type="text/javascript"></script>
<link href='/css/csdn_feed.css?1597309853' rel='stylesheet' />
</head>
<body data-category="home" data-host_type="www">
......
3 解析数据
3.1 安装BeautifulSoup
3.1.1 使用pip安装
pip install beautifulsoup4
3.2 解析数据
3.2.1 引入模块
from bs4 import BeautifulSoup
3.2.2 解析数据
soup = BeautifulSoup(r.text.encode(r.encoding).decode(),"lxml")
print("网页源码：\n"+str(soup)+"\n\n")
运行结果
网页源码：
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<meta content="IE=Edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no" name="viewport"/>
<meta content="black" name="apple-mobile-web-app-status-bar-style"/>
<meta content="always" name="referrer"/>
<meta content="3189512127C34C46BC74BED5852D45E4" name="msvalidate.01"/>
<title>优快云 - 专业开发者社区</title>
<meta content="优快云是全球知名中文IT技术交流平台,创建于1999年,包含原创博客、精品问答、职业培训、技术论坛、资源下载等产品服务,提供原创、优质、完整内容的专业IT技术开发社区." data-hid="description" data-n-head="true" name="description"/>
<script src="//g.csdnimg.cn/tingyun/1.8.3/www.js" type="text/javascript"></script>
<link href="https://www.youkuaiyun.com/" ref="canonical"/>
<link href="//csdnimg.cn/public/favicon.ico" rel="SHORTCUT ICON"/>
<link href="//csdnimg.cn/public/common/toolbar/content_toolbar_css/content_toolbar.css" rel="stylesheet"/>
<link href="//csdnimg.cn/public/common/libs/bootstrap/css/bootstrap.min.css" rel="stylesheet"/>
<link href="//csdnimg.cn/public/static/css/avatar.css" rel="stylesheet"/>
<link href="//g.csdnimg.cn/nav-second/1.0.2/css/nav-second.css" rel="stylesheet" type="text/css"/>
<script src="//csdnimg.cn/public/common/libs/jquery/jquery-1.9.1.min.js" type="text/javascript"></script>
<link href="/css/csdn_feed.css?1597309853" rel="stylesheet"/>
</head>
4 查找指定信息
可以使用findAll来查找HTML代码中某个标签
4.1 查找其中一个数据
我们可以先查找一下li标签，li标签再csdn中出现的比较多：
l=soup.findAll("li")
print("查找到的所有list\n"+str(l)+"\n\n")
运行结果：
查找到的所有list
[<li class="active" data-slide-to="0" data-target="#myCarousel"></li>, <li class="" data-slide-to="1" data-target="#myCarousel"></li>, <li class="" data-slide-to="2" data-target="#myCarousel"></li>, <li class="" data-slide-to="3" data-target="#myCarousel"></li>, <li class="" data-slide-to="4" data-target="#myCarousel"></li>, <li class="tip_box clearfix">
<button class="txt btn-feed-refresh" type="button">刚刚阅读在这里，点击刷新</button>
<div class="read-here csdn-tracking-statistics" data-dsm="post" data-report-click='{"mod":"popu_464"}'>
<a>刚刚阅读在这里，点击刷新</a>
</div>
</li>, <li data-report-click='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/Blockchain_lemon/article/details/108301528","strategy":"","index":"0"}' data-report-view='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/Blockchain_lemon/article/details/108301528","strategy":"","index":"0"}'>
<div class="img_box"><a href="https://blog.youkuaiyun.com/Blockchain_lemon/article/details/108301528" target="_blank"><img alt="" src="https://csdnimg.cn/feed/20200831/99e34ee8e49ca4dc83ad815fc0d8432c.jpg?x-oss-process=image/resize,h_64"/></a></div>
<div class="content">
<h3 class="company_name"><a href="https://blog.youkuaiyun.com/Blockchain_lemon/article/details/108301528" target="_blank" title="以太坊Layer 2 扩容，V神为什么偏爱ZK rollup ？">以太坊Layer 2 扩容，V神为什么偏爱ZK rollup ？</a></h3>
</div>
</li>, <li data-report-click='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/csdnnews/article/details/108313740","strategy":"","index":"1"}' data-report-view='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/csdnnews/article/details/108313740","strategy":"","index":"1"}'>
<div class="img_box"><a href="https://blog.youkuaiyun.com/csdnnews/article/details/108313740" target="_blank"><img alt="" src="https://csdnimg.cn/feed/20200831/d626555c36a954cc529c669c13263ac2.png?x-oss-process=image/resize,h_64"/></a></div>
<div class="content">
<h3 class="company_name"><a href="https://blog.youkuaiyun.com/csdnnews/article/details/108313740" target="_blank" title="代码没写完，哪里有脸睡觉！17 张程序员壁纸推荐">代码没写完，哪里有脸睡觉！17 张程序员壁纸推荐</a></h3>
</div>
</li>, <li data-report-click='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/dQCFKyQDXYm3F8rB0/article/details/108301433","strategy":"","index":"2"}' data-report-view='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/dQCFKyQDXYm3F8rB0/article/details/108301433","strategy":"","index":"2"}'>
<div class="img_box"><a href="https://blog.youkuaiyun.com/dQCFKyQDXYm3F8rB0/article/details/108301433" target="_blank"><img alt="" src="https://csdnimg.cn/feed/20200831/021a2e0fa89baa594f23dde42ac7b339.jpg?x-oss-process=image/resize,h_64"/></a></div>
<div class="content">
<h3 class="company_name"><a href="https://blog.youkuaiyun.com/dQCFKyQDXYm3F8rB0/article/details/108301433" target="_blank" title="史上最火 ECCV 已开幕，这些论文都太有意思了">史上最火 ECCV 已开幕，这些论文都太有意思了</a></h3>
</div>
</li>, <li data-report-click='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/csdnnews/article/details/108313742","strategy":"","index":"3"}' data-report-view='{"mod":"popu_474","dest":"https://blog.youkuaiyun.com/csdnnews/article/details/108313742","strategy":"","index":"3"}'>
<div class="img_box"><a href="https://blog.youkuaiyun.com/csdnnews/article/details/108313742" target="_blank"><img alt="" src="https://csdnimg.cn/feed/20200831/395146c4d836277643af716325a2d193.jpg?x-oss-process=image/resize,h_64"/></a></div>
<div class="content">
<h3 class="company_name"><a href="https://blog.youkuaiyun.com/csdnnews/article/details/108313742" target="_blank" title="英特尔加入 GPU 战局，终用上 6nm 工艺？">英特尔加入 GPU 战局，终用上 6nm 工艺？</a></h3>
</div>
</li>, <li data-report-click='{"mod":"popu_474","dest":"https://developer.aliyun.com/article/771035?utm_content=g_1000177920","strategy":"","index":"4"}' data-report-view='{"mod":"popu_474","dest":"https://developer.aliyun.com/article/771035?utm_content=g_1000177920","strategy":"","index":"4"}'>
<div class="img_box"><a href="https://developer.aliyun.com/article/771035?utm_content=g_1000177920" target="_blank"><img alt="" src="https://csdnimg.cn/feed/20200904/d1a94697e6c461667ff25d1a7557e810.jpg?x-oss-process=image/resize,h_64"/></a></div>
<div class="content">
<h3 class="company_name"><a href="https://developer.aliyun.com/article/771035?utm_content=g_1000177920" target="_blank" title="开放下载！《淘系千人千面分发体系全拆解》全方位视角为你解读阿里云搜索与推荐技术的实践与应用。详情链接：">开放下载！《淘系千人千面分发体系全拆解》全方位视角为你解读阿里云搜索与推荐技术的实践与应用。详情链接：</a></h3>
</div>
</li>
.....
]
4.2 获取里面的数据
上面的带了“li”标签，不过大部分时候我们都是获取标签里的内容
l=[x.text for x in soup.findAll("li")]
print("查找到的所有list的内容\n"+str(l)+"\n\n")
运行结果：
查找到的所有list的内容
['', '', '', '', '', '\n刚刚阅读在这里，点击刷新\n\n刚刚阅读在这里，点击刷新\n\n', '\n\n\n以太坊Layer 2 扩容，V神为什么偏爱ZK rollup ？\n\n', '\n\n\n代码没写完，哪里有脸睡觉！17 张程序员壁纸推荐\n\n', '\n\n\n史上最火 ECCV 已开幕，这些论文都太有意思了\n\n', '\n\n\n英特尔加入 GPU 战局，终用上 6nm 工艺？\n\n', '\n\n\n开放下载！《淘系千人千面分发体系全拆解》全方位视角为你解读阿里云搜索与推荐技术的实践与应用。详情链接：\n\n', '\n\n\nBitcoin SV开发者大会\n比特币区块链技术峰会\n\n', '\n\n\nUnity开发者专区\n9月1日第二期火热开班 仅限50人 报满即止！\n\n', '\n\n\n腾讯云AI 连接智能产业\n人工智能行业应用先驱\n\n', '\n\n\nQualcomm 开发者专区\nQualcomm 开发者专区\n\n', '\n\n\n腾讯位置服务技术专区\n腾讯位置服务技术专区\n\n', '\n\n\n英特尔开发人员专区\n了解全新技术，缩短开发周期，帮助您更快地将产品推向市场。\n\n', '\n\n\n华为开发者专区\n华为开放能力及平台服务\n\n', '\n\n\n总奖金近9万！视频超分辨率大赛等你来战！\n\n', '\n\n\n华为开发者学院\n掌握最新热门技术课程\n\n', '\n\n你期待鸿蒙OS版手机吗？\n', '\n\n禁用MATLAB！哈工大参加全国大学生数学建模竞赛被提特殊要求\n', '\n\n27\n08月\n\n\n10万奖金等你拿！2020第四届易观OLAP算法大赛火热开启\n线上\n\n', '\n\n09\n09月\n\n\nIntel 工业Edge Insights & IOT系列课程\n线上\n\n', '\n\n09\n09月\n\n\n腾讯全球数字生态大会-2020云上会\n线上\n\n', '\n\n11\n09月\n\n\n移动云TeaTalk技术沙龙-云原生技术专场 上海站\n线上\n\n', '\n\n25\n09月\n\n\n高通AI创新应用大赛\n线上\n\n', '\n\n30\n09月\n\n\n5W元大奖等你来，移动云专题赛正式启动！\n线上\n\n', '\n\n\n\n\n华为云官方博客\n\n\n\n\n\n\n\n\n\n\n\n\n\n原创1497\n粉丝14147\n获赞5974\n\n\n', '\n\n\n\n\n阿里技术\n\n\n\n\n\n\n\n\n\n\n\n\n\n原创788\n粉丝7502\n获赞2099\n\n\n', '\n\n\n\n\n美团技术团队\n\n\n\n\n\n\n\n\n\n\n\n\n\n原创501\n粉丝10823\n获赞641\n\n\n', '\n\n\n\n\n巨杉数据库技术社区\n\n\n\n\n\n\n\n\n\n\n\n\n\n原创148\n粉丝449\n获赞32\n\n\n', '\n\n\n\n\n千锋教育\n\n\n\n\n\n\n\n\n\n\n\n\n\n原创663\n粉丝1945\n获赞367\n\n\n', '\n\n\n\n\n百度大脑\n\n\n\n\n\n\n\n\n\n\n\n\n\n原创178\n粉丝453\n获赞41\n\n\n']
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
