Spider-requests库

最新推荐文章于 2024-07-28 03:35:43 发布

u014490943

最新推荐文章于 2024-07-28 03:35:43 发布

阅读量243

点赞数

本文链接：https://blog.youkuaiyun.com/u014490943/article/details/107439719

版权

1-requests库的基本使用

2-requests发送post请求

3-requests使用代理ip

4-requests处理cookie信息

5-requests处理不信任的ssl证书

1-requests库的基本使用

安装

pip install requests

中文文档：

英文文档：http://github.com/requests/requests

import requests

url = 'http://btbtdy.tv/btfl/dy1.html'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
    # 'Referer':'http://www.btbtdy.tv/',
    # 'Cookie':'Hm_lvt_99249fb41a838398a3cc1c3ad2258fe7=1545717237; Hm_lpvt_99249fb41a838398a3cc1c3ad2258fe7=1545718246; PHPSESSID=prj2onjgngpi7p7grfmbea5946; Hm_lvt_99249fb41a838398a3cc1c3ad2258fe7=1545717237; Hm_lpvt_99249fb41a838398a3cc1c3ad2258fe7=1545718538',
}
rep = requests.get(url,headers=headers)
# 返回文本信息
# print(rep.text)
# 返回字节数据，需要转码
print(rep.content.decode('utf-8'))
# 返回URL
# print(rep.url)
# 返回状态码
print(rep.status_code)
# 返回编码类型
# print(response.encoding)

2-requests发送post请求

#encoding: utf-8

import requests

data = {
    'first':"true",
    'pn': '1',
    'kd': 'python'
}
headers = {
    'Referer': 'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36'
}

response = requests.post('https://www.lagou.com/jobs/positionAjax.json?city=%E6%B7%B1%E5%9C%B3&needAddtionalResult=false&isSchoolJob=0',data=data,headers=headers)
# 如果返回的是json数据。那么可以调用`response.json()`来将json字符串转换为字典或者列表。
print(type(response.json()))
print(response.json())

3-requests使用代理ip

#encoding: utf-8
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36'
}
# 代理IP，透明的表示为可以看到原来的IP地址，所以选择高匿的
proxy = {
    'http': '123.54.194.96:38661'
}

response = requests.get("http://httpbin.org/ip",headers=headers ,proxies=proxy)
print(response.text)

4-requests处理cookie信息

#encoding: utf-8

import requests

# 1.
# 使用 get_dict() 方法会返回一个字典信息
# response = requests.get('https://www.baidu.com/')
# print(response.cookies.get_dict())


# 2.
url = "http://www.renren.com/PLogin.do"
data = {"email":"970138074@qq.com",'password':"pythonspider"}
headers = {
    'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
}

session = requests.Session()
# 所有的操作都在session上进行
session.post(url,data=data,headers=headers)
# 登录页面
response = session.get('http://www.renren.com/880151247/profile')
with open('renren.html','w',encoding='utf-8') as fp:
    fp.write(response.text)

5-requests处理不信任的ssl证书

#encoding: utf-8
import requests

# 添加参数verify，不需要验证证书，处理不信任的SSL证书
resp = requests.get('http://www.12306.cn/mormhweb/',verify=True)
print(resp.encoding)
with open('12306.html','w',encoding='ISO-8859-1') as fp:
    fp.write(resp.text)

返回顶部