python爬虫

最新推荐文章于 2024-02-13 10:30:42 发布

k0sec

最新推荐文章于 2024-02-13 10:30:42 发布

阅读量221

点赞数

分类专栏： python 文章标签： python

本文链接：https://blog.youkuaiyun.com/ghx13630444675/article/details/91488721

版权

python 专栏收录该内容

4 篇文章

订阅专栏

GET方式

>>> import requests
>>> html_str = requests.get('http://exercise.kingname.info/exercise_requests_get.html').content.decode()
>>> print(html_str)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>GET 测试</title>
</head>
<body>
<p>如果你可以看到这一段话，那么说明你已经知道如何通过</p>
<p>requests获得GET方式网页的源代码。</p>
</body>
</html>
>>>

POST方式
提交formdata数据

>>> import requests
>>> data = {'name':'kingname','password':'1234567'}
>>> html_formdata = requests.post('http://exercise.kingname.info/exercise_requests_post',data = data).content.decode()
>>> print(html_formdata)



<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>POST 测试</title>
</head>
<body>

<p>恭喜恭喜，你现在使用的是POST方式访问本页，通过Formdata提交数据，你提交的数据为：</p>
<p>Name: kingname</p>
<p>Password: 1234567</p>

</body>
</html>
>>>

提交JSON数据

>>> import requests
>>> data = {'name':'kingname','password':'1234567'}
>>> html_formdata = requests.post('http://exercise.kingname.info/exercise_requests_post',json = data).content.decode()
>>> print(html_formdata)



<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>POST 测试</title>
</head>
<body>

<p>恭喜恭喜，你现在使用的是POST方式访问本页，通过JSON提交数据，你提交的数据为：</p>
<p>Name: kingname</p>
<p>Password: 1234567</p>

</body>
</html>
>>>

requests与正则表达式
提取标题

title = re.search('title>(.*?)<',html,re.S).group(1)

提取正文

content_list = re.findall('<p>(.*?)<',html,re.S)
content_str = '\n'.join(content_list)

>>> import requests
>>> import re
>>>
>>> html = requests.get('http://exercise.kingname.info/exercise_requests_get.html').content.decode()
>>> title = re.search('title>(.*?)<',html,re.S).group(1)
>>> content_list = re.findall('<p>(.*?)<',html,re.S)
>>> content_str = '\n'.join(content_list)
>>> print(f'页面标题为:{title}')
页面标题为:GET 测试
>>> print(f'页面正文内容为:\n{content_str}')
页面正文内容为:
如果你可以看到这一段话，那么说明你已经知道如何通过
requests获得GET方式网页的源代码。
>>>

多线程爬虫

>>> from multiprocessing.dummy import Pool
>>> def calc_power2(num):
...     return num * num
>>> pool = Pool(3)
>>> origin_num = [x for x in range(10)]
>>> result = pool.map(calc_power2,origin_num)
>>> print(f'计算0-9的平方分别为:{result}')
计算0-9的平方分别为:[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>

单线程访问100次百度首页

import requests
import time
from multiprocessing.dummy import Pool

def query(url):
     requests.get(url)

start = time.time()
for i in range(100):
    query('https://www.baidu.com')

end = time.time()
print(f'单线程循环访问100次百度首页，耗时:{end-start}')

多线程访问100次百度首页

import requests
import time
from multiprocessing.dummy import Pool

def query(url):
    requests.get(url)


start = time.time()
url_list = []
for i in range(100):
    url_list.append('https://baidu.com')
pool = Pool(5)
pool.map(query,url_list)
end = time.time()
print(f'5线程访问100次百度首页，耗时:{end-start}')