【爬虫基础】Requests各种用法_request proxies-优快云博客

标题

一、GET请求

1. 基础请求

import requests
 
response = requests.get('http://httpbin.org/get')
print(response.text)

2. 携带参数

方式一：

import requests
 
response = requests.get('http://httpbin.org/get?name=0bug&age=25')
print(response.text)

方式二：

import requests
 
data = {
    'name': '0bug',
    'age': 25
}
response = requests.get('http://httpbin.org/get', params=data)
print(response.text)```

#### 3. 

### 二、POST请求
```python
import requests
 
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36'
 
}
response = requests.get('https://www.baidu.com/', headers=headers)
print(response.status_code)

3. Json数据

import requests
 
response = requests.get('http://httpbin.org/get')
print(type(response.text))
print(response.json())
print(type(response.json()))

4. 二进制数据

import requests
 
response = requests.get('https://github.com/favicon.ico')
print(type(response.text))
print(type(response.content))
print(response.text)
print(response.content)

5. 二进制数据

import requests
 
response = requests.get('https://github.com/favicon.ico')
print(type(response.text))
print(type(response.content))
print(response.text)
print(response.content)

6. 图片处理

import requests
 
response = requests.get('https://github.com/favicon.ico')
with open('img.ico','wb') as f:
    f.write(response.content)

7. Headers

import requests
 
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36'
 
}
response = requests.get('https://www.baidu.com/', headers=headers)
print(response.status_code)

二、POST请求

1. 基础请求

import requests
 
data = {'name':'0bug'}
response = requests.post('http://httpbin.org/post',data=data)
print(response.text)

2. data参数

import requests
 
data = {'name': '0bug'}
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36'
 
}
response = requests.post('http://httpbin.org/post', data=data, headers=headers)
print(response.json())

3.response属性

import requests
 
response = requests.get('http://www.jianshu.com')
print(type(response.status_code), response.status_code)
print(type(response.headers), response.headers)
print(type(response.cookies), response.cookies)
print(type(response.url), response.url)
print(type(response.history), response.history)

4. 文件上传

import requests
 
response = requests.get('http://www.jianshu.com')
print(type(response.status_code), response.status_code)
print(type(response.headers), response.headers)
print(type(response.cookies), response.cookies)
print(type(response.url), response.url)
print(type(response.history), response.history)

5. 获取cookie

import requests
 
response = requests.get('https://www.baidu.com')
print(response.cookies)
for key,value in response.cookies.items():
    print(key+'='+value)

6. 获取cookie

import requests
 
response = requests.get('https://www.baidu.com')
print(response.cookies)
for key,value in response.cookies.items():
    print(key+'='+value)

三、会话保持

1. Session

import requests
 
s = requests.Session()
s.get('http://httpbin.org/cookies/set/number/123456')
response = s.get('http://httpbin.org/cookies')
print(response.text)

四、证书验证

1. 无证书报错

import requests
 
response = requests.get('https://www.12306.cn')
print(response.status_code)

2. 无证书验证

import requests
 
response = requests.get('https://www.12306.cn',verify=False)
print(response.status_code)

3. 消除警告信息

import requests
import urllib3
urllib3.disable_warnings()
response = requests.get('https://www.12306.cn',verify=False)
print(response.status_code)

4. 使用本地证书

import requests
 
response = requests.get('https://www.12306.cn',cert=('/path/server.crt','/path/key'))
print(response.status_code)

五、代理设置

1. 基础设置

import requests
 
proxies = {
    'http': 'http://127.0.0.1:9743',
    'https': 'https://127.0.0.1:9743'
}
 
response = requests.get('https://www.taobao.com', proxies=proxies)
print(response.status_code)

2. 带账户和密码


import requests
 
proxies = {
    'http': 'http://127.0.0.1:9743',
    'https': 'https://127.0.0.1:9743'
}
 
response = requests.get('https://www.taobao.com', proxies=proxies)
print(response.status_code)

3. socks代理

pip install requests[socks]
import requests
 
proxies = {
    'http': 'socks5://127.0.0.1:9742',
    'https': 'socks5://127.0.0.1:9742',
}
 
response = requests.get('https://www.taobao.com', proxies=proxies)
print(response.status_code)

六、超时设置

1. 超时设置

import requests
 
response = requests.get('https://www.taobao.com',timeout=0.01)
print(response.status_code)

2. 错误处理

import requests
from requests.exceptions import ReadTimeout
 
try:
    response = requests.get('https://www.taobao.com', timeout=0.01)
    print(response.status_code)
except ReadTimeout:
    print('time out')

七. 认证设置

方式一：

import requests
from requests.auth import HTTPBasicAuth
 
r = requests.get('http://127.0.0.1:8080', auth=HTTPBasicAuth('user', '123'))
print(r.status_code)

方式二：

import requests
 
r = requests.get('http://127.0.0.1:8080', auth=('user', '123'))
print(r.status_code)

八、异常处理

1.异常处理

import requests
from requests.exceptions import ReadTimeout, HTTPError, RequestException
 
try:
    response = requests.get('http://httpbin.org/get', timeout=0.01)
    print(response.status_code)
except ReadTimeout:
    print('Timeout')
except HTTPError:
    print('HTTP REEOR')
except RequestException:
    print('Error')