HTTP status code is not handled or not allowed的解决方法

最新推荐文章于 2022-04-29 21:52:55 发布
原创 最新推荐文章于 2022-04-29 21:52:55 发布 · 4.1w 阅读 · 12 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。 

/Books/>: HTTP status code is not handled or not allowed
2017-11-04 17:21:38 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://www.dmoz.org/Computers/Programming/Languages/Python
/Resources/>: HTTP status code is not handled or not allowed

我遇到的这个问题出现在scrapy里面，解决办法是在settings里面添加

HTTPERROR_ALLOWED_CODES = [403]#上面报的是403，就把403加入。

彩蛋：

scrapy默认是遵守爬虫准则的，即settings里面，ROBOTSTXT_OBEY = True。
比如抓取百度，在https://www.baidu.com/robots.txt里，有这样的一个规范。如果遵守，比如今日头条，是不能用scrapy爬取的。这个时候需要把ROBOTSTXT_OBEY=False.也就是不遵守它的规则。

CEF框架：各种各样的Handle(三)——拦截Http的请求与响应
新兴IT民工的专栏
05-24 1140
前面有两篇提到了CEF框架中的各种各样的HANDLE，然后中间一段时间关注NodeJS去了，今天再补充一点CEF的东西：利用CEF框架中的各种Handle来拦截Http请求与其响应。也还是基于CEF示例代码中的CEFSIMPLE来说明。
scrapy 出现INFO: Ignoring response ＜500 xxx＞: HTTP status code is not handled or not allowed
哈哈儿 博客
05-10 522
scrapy 出现INFO: Ignoring response <500 xxx>: HTTP status code is not handled or not allowed 我寻思500是服务器错误，肯定是没有数据返回的。但是： 本机使用curl xxx是有数据返回的。 使用requests.get模块也是有数据的，然后打印了下status_code发现是500. 遂在settings中添加了HTTPERROR_ALLOWED_CODES = [500]发现是ok的。 ...
scrapy 解决Redirecting 301 302重定向问题
热门推荐
菜鸡小白的成长记录
10-25 4万+
在使用Scrapy框架中URl被重定向，总是遇到这类问题: DEBUG: Redirecting (301/302) to <GET https://XXXX refer https://XXXX> 解决方式： 在Scrapy中的Request中添加 dont_filter=True，因为Scrapy是默认过滤掉重复的请求URL，添加上参数之后即使被重定向了也能请求到正常的数据了。 在Scrapy框架中的 settings.py文件里添加 HTTPERROR_ALLOWED_COD
python初步学习中的问题之HTTP status code is not handled or not allowed
aov35990的博客
04-05 876
按照网上的教程学习python，最开始的dmoz_spider项目一直不能正常运行，一直出现下面的问题： 即http status code is not handled or allowed http状态代码没有被处理或允许， 一开始我不理解，在网上找了好多修改setting.py中的DOWNLOADER_MIDDLEWARES方法也没啥用，心疼自己英语太差，好多问题看不懂。...
HTTP status code is not handled or not allowed
weixin_45656577的博客
01-18 2144
用scrapy爬取知乎的时候 报错: 403 HTTP status code is not handled or not allowed 替换setting.py的默认headers无效, 直接使用User_Agent无效, 在中间件中 指定UA中间件无效 解决: 在setting.py里面加入: HTTPERROR_ALLOWED_CODES = [403] ...
上面的代码运行后在cmd中显示<403https://www.tianqi.com/>:HTTP status code is not handled or not allowed如何解决
06-09
要解决这个问题，你需要在代码中添加处理状态码的逻辑，比如当收到 403 响应时，可以打印出错误信息或者重新发起请求等。具体的处理方法取决于你的代码实现方式和需求。 另外，你也可以检查一下请求头是否正确，...
raise KeyError(f"{self.__class__.__name__} does not support field: {key}") KeyError: 'NepuContentItem does not support field: publish_date' 2025-07-07 20:46:23 [nepudata] INFO: 开始解析页面: https://www.nepu.edu.cn/gjjl/xjjl.htm 2025-07-07 20:46:23 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/gjjl/xjjl.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\scrapy\utils\defer.py", line 257, in iter_errback yield next(it) ^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\utils\python.py", line 312, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\utils\python.py", line 312, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync for r in iterable: File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 28, in <genexpr> return (r for r in result or () if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync for r in iterable: File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\referer.py", line 353, in <genexpr> return (self._set_referer(r, response) for r in result or ()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync for r in iterable: File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 27, in <genexpr> return (r for r in result or () if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync for r in iterable: File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\depth.py", line 31, in <genexpr> return (r for r in result or () if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync for r in iterable: File "C:\Users\Lenovo\nepu_spider\nepu_spider\spiders\nepudata_spider.py", line 93, in parse item = self.extract_content(response) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Lenovo\nepu_spider\nepu_spider\spiders\nepudata_spider.py", line 175, in extract_content item['publish_date'] = publish_date ~~~~^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\item.py", line 85, in __setitem__ raise KeyError(f"{self.__class__.__name__} does not support field: {key}") KeyError: 'NepuContentItem does not support field: publish_date' 2025-07-07 20:46:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/yjsy/>: HTTP status code is not handled or not allowed 2025-07-07 20:46:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/jwc/>: HTTP status code is not handled or not allowed 2025-07-07 20:46:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsc/>: HTTP status code is not handled or not allowed 2025-07-07 20:46:25 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/rsc/>: HTTP status code is not handled or not allowed 2025-07-07 20:46:26 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/kyc/>: HTTP status code is not handled or not allowed 2025-07-07 20:46:26 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2020>: HTTP status code is not handled or not allowed 2025-07-07 20:46:27 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2021>: HTTP status code is not handled or not allowed 2025-07-07 20:46:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2022>: HTTP status code is not handled or not allowed 2025-07-07 20:46:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2023>: HTTP status code is not handled or not allowed 2025-07-07 20:46:29 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2024>: HTTP status code is not handled or not allowed 2025-07-07 20:46:29 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2025>: HTTP status code is not handled or not allowed 2025-07-07 20:46:30 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2020>: HTTP status code is not handled or not allowed 2025-07-07 20:46:30 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2021>: HTTP status code is not handled or not allowed 2025-07-07 20:46:31 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2022>: HTTP status code is not handled or not allowed 2025-07-07 20:46:31 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2023>: HTTP status code is not handled or not allowed 2025-07-07 20:46:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2024>: HTTP status code is not handled or not allowed 2025-07-07 20:46:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2025>: HTTP status code is not handled or not allowed 2025-07-07 20:46:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2020>: HTTP status code is not handled or not allowed 2025-07-07 20:46:34 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2021>: HTTP status code is not handled or not allowed 2025-07-07 20:46:34 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2022>: HTTP status code is not handled or not allowed 2025-07-07 20:46:35 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2023>: HTTP status code is not handled or not allowed 2025-07-07 20:46:36 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2024>: HTTP status code is not handled or not allowed 2025-07-07 20:46:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2025>: HTTP status code is not handled or not allowed 2025-07-07 20:46:37 [scrapy.core.engine] INFO: Closing spider (finished) 2025-07-07 20:46:37 [nepudata] INFO: 数据库连接已关闭 2025-07-07 20:46:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 10178, 'downloader/request_count': 32, 'downloader/request_method_count/GET': 32, 'downloader/response_bytes': 117980, 'downloader/response_count': 32, 'downloader/response_status_count/200': 9, 'downloader/response_status_count/404': 23, 'elapsed_time_seconds': 23.999904, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 7, 7, 12, 46, 37, 62452), 'httpcompression/response_bytes': 187569, 'httpcompression/response_count': 9, 'httperror/response_ignored_count': 23, 'httperror/response_ignored_status_count/404': 23, 'log_count/ERROR': 9, 'log_count/INFO': 45, 'log_count/WARNING': 1, 'response_received_count': 32, 'scheduler/dequeued': 32, 'scheduler/dequeued/memory': 32, 'scheduler/enqueued': 32, 'scheduler/enqueued/memory': 32, 'spider_exceptions/KeyError': 9, 'start_time': datetime.datetime(2025, 7, 7, 12, 46, 13, 62548)} 2025-07-07 20:46:37 [scrapy.core.engine] INFO: Spider closed (finished)
最新发布
07-08
### 解决方法 #### 1. 确保字段已在 `Item` 类中定义 首先检查你的 `Item` 类（例如 `NepuContentItem`）是否已经定义了 `publish_date` 字段： ```python import scrapy class NepuContentItem(scrapy.Item): ...
import scrapy class WeatherSpider(scrapy.Spider): name = 'weather' allowed_domains = ['tianqi.com'] start_urls = ['https://www.tianqi.com/'] def parse(self, response): city = 'beijing' # 自选城市 url = f'https://tianqi.com/{city}/' yield scrapy.Request(url, callback=self.parse_weather) def parse_weather(self, response): date = response.css('.week>p::text').get() weather = response.css('.weather span::text').get() temperature = response.css('.weather .temp::text').get() humidity = response.css('.weather .shidu::text').get() wind = response.css('.weather .zi .name::text').get() with open('weather.txt', 'a', encoding='utf-8') as f: f.write(f'{date} {weather} {temperature} {humidity} {wind}\n'上面的代码运行后在cmd中显示<403https://www.tianqi.com/>:HTTP status code is not handled or not allowed如何解决)
06-09
在你的代码中，你请求的 URL 是 https://tianqi.com/beijing/，而不是 https://www.tianqi.com/，这可能是引起 403 错误的原因之一。 此外，你需要在请求头中添加一个 User-Agent 字段，模拟一个浏览器的请求，以...
2025-07-06 23:19:30 [scrapy.extensions.telnet] INFO: Telnet Password: d45adb2d404b9cba 2025-07-06 23:19:30 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.throttle.AutoThrottle'] 2025-07-06 23:19:30 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2025-07-06 23:19:30 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-07-06 23:19:30 [scrapy.middleware] INFO: Enabled item pipelines: ['nepu_spider.pipelines.ContentCleanPipeline', 'nepu_spider.pipelines.DeduplicatePipeline', 'nepu_spider.pipelines.SQLServerPipeline'] 2025-07-06 23:19:30 [scrapy.core.engine] INFO: Spider opened 2025-07-06 23:19:30 [info] INFO: ✅ 成功连接到 SQL Server 数据库 2025-07-06 23:19:30 [info] INFO: ✅ 数据库连接测试通过 2025-07-06 23:19:30 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-07-06 23:19:30 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2025-07-06 23:19:30 [info] INFO: 🚀 开始爬取东北石油大学官网... 2025-07-06 23:19:30 [info] INFO: 初始URL数量: 14 2025-07-06 23:19:30 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/tzgg.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:33 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/xxgk/xxjj.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:36 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/jgsz/dzglbm.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:38 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/jgsz/xysz.htm>: HTTP status code is not handled or not allowed 2025-07-06 23:19:39 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/rcpy1.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:40 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/kxyj/kygk1.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:40 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/szdw/szgk.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:41 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/zsjy1.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:42 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/xtgz1.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:42 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/gjjl/xjjl.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:43 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xyfg.htm>: HTTP status code is not handled or not allowed 2025-07-06 23:19:44 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsc/xsgz.htm>: HTTP status code is not handled or not allowed 2025-07-06 23:19:44 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/xxgk/xrld.htm> (referer: None) Traceback (most recent call last): File "D:\annaCONDA\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\annaCONDA\Lib\site-packages\scrapy\spiders\__init__.py", line 76, in parse raise NotImplementedError( NotImplementedError: InfoSpider.parse callback is not defined 2025-07-06 23:19:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/jxky/jwtz.htm>: HTTP status code is not handled or not allowed 2025-07-06 23:19:45 [scrapy.core.engine] INFO: Closing spider (finished) 2025-07-06 23:19:45 [info] INFO: ✅ 数据库连接已关闭 2025-07-06 23:19:45 [info] INFO: 🛑 爬虫结束，原因: finished 2025-07-06 23:19:45 [info] INFO: 总计爬取页面: 0 2025-07-06 23:19:45 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 3312, 'downloader/request_count': 14, 'downloader/request_method_count/GET': 14, 'downloader/response_bytes': 74925, 'downloader/response_count': 14, 'downloader/response_status_count/200': 10, 'downloader/response_status_count/404': 4, 'elapsed_time_seconds': 14.564797, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 7, 6, 15, 19, 45, 288206), 'httpcompression/response_bytes': 188892, 'httpcompression/response_count': 10, 'httperror/response_ignored_count': 4, 'httperror/response_ignored_status_count/404': 4, 'log_count/ERROR': 10, 'log_count/INFO': 21, 'log_count/WARNING': 1, 'response_received_count': 14, 'scheduler/dequeued': 14, 'scheduler/dequeued/memory': 14, 'scheduler/enqueued': 14, 'scheduler/enqueued/memory': 14, 'spider_exceptions/NotImplementedError': 10, 'start_time': datetime.datetime(2025, 7, 6, 15, 19, 30, 723409)} 2025-07-06 23:19:45 [scrapy.core.engine] INFO: Spider closed (finished)
07-07
### 三级标题：Scrapy 中 `Spider.parse callback is not defined` 错误的原因与解决方法 在使用 Scrapy 框架开发爬虫时，若出现 `NotImplementedError: InfoSpider.parse callback is not defined` 错误，通常表明...
爬取知乎碰到的问题-------------------------1、HTTP status code is not handled or not allowed
replat-xin
01-16 2401
HTTP status code is not handled or not allowed 解决： 只需在setting中添加 HTTPERROR_ALLOWED_CODES = [错误码]  
Scrapy爬虫报错HTTP status code is not handled or not allowed
weixin_50197124的博客
04-29 1140
爬取网页显示403HTTP status code is not handled or not allowed 解决方法： 在settings.py里把代理关掉： DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, }
解决HTTP status code is not handled or not allowed
weixin_30596735的博客
03-09 3468
/Books/>: HTTP status code is not handled or not allowed 2017-11-04 17:21:38 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://www.dmoz.org/Computers/Programming/Langua...
Scrapy爬虫小技巧02：HTTP status code is not handled or not allowed的解决方法
weixin_41961559的博客
04-08 6365
在scrapy爬虫的过程中出现 HTTP status code is not handled or not allowed 的问题导致爬虫无法继续，如下截图： 解决方式： 在settings.py文件中添加： HTTPERROR_ALLOWED_CODES = [302] 截图中报错302，就在括号里添加302。若报错403就添加403。 附加说明： scrapy框架中有许多默认...
scrapy中 HTTP status code is not handled or not allowed异常处理
u013109501的博客
08-20 8593
scrapy中的setting文件中添加 HTTPERROR_ALLOWED_CODES = [403]
scrapy爬取报HTTP status code is not handled or not allowed的可能解决方法
aGreySky的博客
03-19 2641
今天写爬虫时遇到了如题的问题，网上的解决方法都是修改get或post请求。但对我都没有用，因为我用postman测了下，get、post请求都能访问到。 网上找了很久，终于找到解决方法——在settings.py中关闭代理，如下即可 DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.useragent.UserAgent...
评论 6
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值