Python requests“Max retries exceeded with url” error

最新推荐文章于 2025-05-20 11:07:39 发布

转载最新推荐文章于 2025-05-20 11:07:39 发布 · 1.7k 阅读

文章标签：

#python连接

本文介绍了解决Python网络请求超时问题的方法，包括增加重试次数、关闭多余连接等，并提供了使用代理访问受限网站的技巧。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

error1:

NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000038F2B00>: Failed to establish a new connection:[WinError 10060] 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败。',))

解决办法：

session.keep_alive=False

error2:

python hostname doesn't match either of facebookXXXXX

解决办法：

importssl

ssl.match_hostname =lambdacert, hostname:True

多方查阅后发现了解决问题的原因：http连接太多没有关闭导致的。

解决办法：

1、增加重试连接次数

requests.adapters.DEFAULT_RETRIES = 5

2、关闭多余的连接

requests使用了urllib3库，默认的http connection是keep-alive的，requests设置False关闭。

s = requests.session()

s.keep_alive =False

3、只用session进行操作。即只创建一个连接，并设置最大连接数或者重试次数。

import requests

from requests.adapters import HTTPAdapter

from requests.packages.urllib3.util.retry import Retry

session = requests.Session()

retry = Retry(connect=3, backoff_factor=0.5)

adapter = HTTPAdapter(max_retries=retry)

session.mount('http://', adapter)

session.mount('https://', adapter)

session.get(url)

import requests

from requests.adapters import HTTPAdapter

from requests.packages.urllib3.util.retry import Retry

s = requests.Session()

retry = Retry(connect =5, backoff_factor =1)

adapter = HTTPAdapter(max_retries = retry)

s.mount('http://', adapter)

s.keep_alive =False

res = s.post(self.conn.host +'/sign-in', data = json.dumps({'name':"XXX",'pwd':"XXX"}))

response = res.json()

但是在starkoverflow上有人给出了这样的解释。

4.安装 py

pip install -U pyopenssl

5、设定固定的睡眠时间在发送请求之间

https://github.com/requests/requests/issues/4246#event

https://stackoverflow.com/questions/23013220/max-retries-exceeded-with-url

在爬取boss直聘时出现这种错误，总结如下：

1.http连接太多没有关闭导致的，解决方法：

import requests

requests.adapters.DEFAULT_RETRIES =5 # 增加重连次数

s = requests.session()

s.keep_alive = False # 关闭多余连接

s.get(url)# 你需要的网址

2.访问次数频繁，被禁止访问，解决方法：使用代理

import requests

s = requests.session()

url ="https://mail.163.com/"

s.proxies= {"https":"47.100.104.247:8080","http":"36.248.10.47:8080", }

s.headers= header

s.get(url)

查找代理的网址：http://ip.zdaye.com/shanghai_ip.html#Free

使用代理时需注意：1.代理分为http和https两种，不能用混，如果把http的代理用作https也是会报上面的错误;2.上面的代理以字典格式传入，例如上面的例子，可以是“47.100.104.247:8080”这种格式，也可以是“https://47.100.104.247:8080”这种格式；3.如果代理不可用一样会报上面的错误。以下方法判断代理是否可用：

import requests

s = requests.session()

url ="https://mail.163.com/"

s.keep_alive = False

s.proxies= {"https":"47.100.104.247:8080","http":"36.248.10.47:8080", }

s.headers= header

r = s.get(url)

print(r.status_code) # 如果代理可用则正常访问，不可用报以上错误

升级

pip install --upgrade requests

如果同一ip访问次数过多也会封ip，这里就要用代理了proxies，python很简单，直接在请求中带上proxies参数就行，

r = requests.get(url, headers=headers, cookies=cookies,proxies = proxies)

代理ip的话，给大家推荐个网站

http://www.data5u.com/

最下方会有20个免费的，一般小爬虫够用了，使用代理就会出现代理连接是否通之类的问题，需要在程序中添加下面的代码，设置连接时间

requests.adapters.DEFAULT_RETRIES =5

s = requests.session()

s.keep_alive = False

from bs4 import BeautifulSoup

import json,requests,sys

reload(sys)

sys.setdefaultencoding('utf-8')

list =[22711693,24759450,69761921,69761921,22743334,66125712,22743270,57496584,75153221,57641884,66061653,70669333,57279088,24740739,66126129,75100027,92667587,92452007,72345827,90004047,90485109,90546031,83527455,91070982,83527745,94273474,80246564,83497073,69027373,96191554,96683472,90500524,92454863,92272204,70443082,96076068,91656438,75633029,96571687,97659144,69253863,98279207,90435377,70669359,96403354,83618952,81265224,77365611,74592526,90479676,56540304,37924067,27496773,56540319,32571869,43611843,58612870,22743340,67293664,67292945, 57641749,75157068,58934198,75156610,59081304,75156647,75156702,67293838,]

returnList = []

proxies = {

# "https": "http://14.215.177.73:80", "http": "http://202.108.2.42:80",

}

headers = {

'Host': 'www.dianping.com',

'Referer': 'http://www.dianping.com/shop/22711693',

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/535.19',

'Accept-Encoding': 'gzip'}

cookies = {

'_lxsdk_cuid': '16146a366a7c8-08cd0a57dad51b-32637402-fa000-16146a366a7c8',

'lxsdk': '16146a366a7c8-08cd0a57dad51b-32637402-fa000-16146a366a7c8',

'_hc.v': 'ec20d90c-0104-0677-bf24-391bdf00e2d4.1517308569',

's_ViewType': '10',

'cy': '16',

'cye': 'wuhan',

'_lx_utm': 'utm_source%3DBaidu%26utm_medium%3Dorganic',

'_lxsdk_s': '1614abc132e-f84-b9c-2bc%7C%7C34'}

requests.adapters.DEFAULT_RETRIES = 5

s = requests.session()

s.keep_alive = False

for i in list:

url = "https://www.dianping.com/shop/%s/review_all" % i

r = requests.get(url, headers=headers, cookies=cookies,proxies = proxies)

# print r.text soup = BeautifulSoup(r.text, 'lxml')

lenth = soup.find_all(class_='PageLink').__len__() + 1

#print lenth for j in xrange(lenth):

urlIn = "http://www.dianping.com/shop/%s/review_all/p%s" % (i, j)

re = requests.get(urlIn, headers=headers, cookies=cookies,proxies =proxies)

soupIn = BeautifulSoup(re.text, 'lxml')

title = soupIn.title.string[0:15]

#print title coment = []

coment = soupIn.select('.reviews-items li')

for one in coment:

try:

if one['class'][0]=='item':

continue

except(KeyError),e:

pass

name = one.select_one('.main-review .dper-info .name')

#print name.get_text().strip() name = name.get_text().strip()

star = one.select_one('.main-review .review-rank span')

#print star['class'][1][7:8] star = star['class'][1][7:8]

pl = one.select_one('.main-review .review-words')

pl['class'] = {'review-words'}

words = pl.get_text().strip()

returnList.append([title,name,star,words])file = open("/Users/huojian/Desktop/store_shop.sql","w")for one in returnList:

file.write("\n")

file.write(unicode(one[0]))

file.write("\n")

file.write(unicode(one[1]))

file.write("\n")

file.write(unicode(one[2]))

file.write("\n")

file.write(unicode(one[3]))

file.write("\n")