python爬虫错误:http.client.HTTPException: got more than 100 headers的解决方法

最新推荐文章于 2025-02-27 14:03:03 发布

Jonariguez

最新推荐文章于 2025-02-27 14:03:03 发布

阅读量2.1k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： Python 技术文章标签： python 爬虫

本文链接：https://blog.youkuaiyun.com/u013068502/article/details/89329073

Python 同时被 2 个专栏收录

5 篇文章

订阅专栏

技术

3 篇文章

订阅专栏

当遇到Python爬虫报错http.client.HTTPException: got more than 100 headers时，原因是服务器返回的请求头数量超过了默认限制。解决方法是修改最大请求头个数。

python爬虫错误:http.client.HTTPException: got more than 100 headers的解决方法

Traceback (most recent call last):
  File "C:\Users\Jonariguez\AppData\Roaming\Python\Python36\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Users\Jonariguez\AppData\Roaming\Python\Python36\site-packages\urllib3\connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "C:\Users\Jonariguez\AppData\Roaming\Python\Python36\site-packages\urllib3\connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "D:\Software\Python3\lib\http\client.py", line 1331, in getresponse
    response.begin()
  File "D:\Software\Python3\lib\http\client.py", line 321, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "D:\Software\Python3\lib\http\client.py", line 211, in parse_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers

错误

返回的请求头大多，超过了自定义的最大请求头个数

解决方法：

修改最大请求头个数即可。

python3

import http.client

http.client._MAXHEADERS = 1000

python2

import httplib

httplib._MAXHEADERS = 1000