raise KeyError(f"{self.__class__.__name__} does not support field: {key}")
KeyError: 'NepuContentItem does not support field: publish_date'
2025-07-07 20:46:23 [nepudata] INFO: 开始解析页面: https://www.nepu.edu.cn/gjjl/xjjl.htm
2025-07-07 20:46:23 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.nepu.edu.cn/gjjl/xjjl.htm> (referer: None)
Traceback (most recent call last):
File "D:\annaCONDA\Lib\site-packages\scrapy\utils\defer.py", line 257, in iter_errback
yield next(it)
^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\utils\python.py", line 312, in __next__
return next(self.data)
^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\utils\python.py", line 312, in __next__
return next(self.data)
^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync
for r in iterable:
File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 28, in <genexpr>
return (r for r in result or () if self._filter(r, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync
for r in iterable:
File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\referer.py", line 353, in <genexpr>
return (self._set_referer(r, response) for r in result or ())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync
for r in iterable:
File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 27, in <genexpr>
return (r for r in result or () if self._filter(r, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync
for r in iterable:
File "D:\annaCONDA\Lib\site-packages\scrapy\spidermiddlewares\depth.py", line 31, in <genexpr>
return (r for r in result or () if self._filter(r, response, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\core\spidermw.py", line 104, in process_sync
for r in iterable:
File "C:\Users\Lenovo\nepu_spider\nepu_spider\spiders\nepudata_spider.py", line 93, in parse
item = self.extract_content(response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Lenovo\nepu_spider\nepu_spider\spiders\nepudata_spider.py", line 175, in extract_content
item['publish_date'] = publish_date
~~~~^^^^^^^^^^^^^^^^
File "D:\annaCONDA\Lib\site-packages\scrapy\item.py", line 85, in __setitem__
raise KeyError(f"{self.__class__.__name__} does not support field: {key}")
KeyError: 'NepuContentItem does not support field: publish_date'
2025-07-07 20:46:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/yjsy/>: HTTP status code is not handled or not allowed
2025-07-07 20:46:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/jwc/>: HTTP status code is not handled or not allowed
2025-07-07 20:46:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsc/>: HTTP status code is not handled or not allowed
2025-07-07 20:46:25 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/rsc/>: HTTP status code is not handled or not allowed
2025-07-07 20:46:26 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/kyc/>: HTTP status code is not handled or not allowed
2025-07-07 20:46:26 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2020>: HTTP status code is not handled or not allowed
2025-07-07 20:46:27 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2021>: HTTP status code is not handled or not allowed
2025-07-07 20:46:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2022>: HTTP status code is not handled or not allowed
2025-07-07 20:46:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2023>: HTTP status code is not handled or not allowed
2025-07-07 20:46:29 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2024>: HTTP status code is not handled or not allowed
2025-07-07 20:46:29 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xxyw/?year=2025>: HTTP status code is not handled or not allowed
2025-07-07 20:46:30 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2020>: HTTP status code is not handled or not allowed
2025-07-07 20:46:30 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2021>: HTTP status code is not handled or not allowed
2025-07-07 20:46:31 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2022>: HTTP status code is not handled or not allowed
2025-07-07 20:46:31 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2023>: HTTP status code is not handled or not allowed
2025-07-07 20:46:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2024>: HTTP status code is not handled or not allowed
2025-07-07 20:46:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/tzgg/?year=2025>: HTTP status code is not handled or not allowed
2025-07-07 20:46:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2020>: HTTP status code is not handled or not allowed
2025-07-07 20:46:34 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2021>: HTTP status code is not handled or not allowed
2025-07-07 20:46:34 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2022>: HTTP status code is not handled or not allowed
2025-07-07 20:46:35 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2023>: HTTP status code is not handled or not allowed
2025-07-07 20:46:36 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2024>: HTTP status code is not handled or not allowed
2025-07-07 20:46:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.nepu.edu.cn/xsdt/?year=2025>: HTTP status code is not handled or not allowed
2025-07-07 20:46:37 [scrapy.core.engine] INFO: Closing spider (finished)
2025-07-07 20:46:37 [nepudata] INFO: 数据库连接已关闭
2025-07-07 20:46:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 10178,
'downloader/request_count': 32,
'downloader/request_method_count/GET': 32,
'downloader/response_bytes': 117980,
'downloader/response_count': 32,
'downloader/response_status_count/200': 9,
'downloader/response_status_count/404': 23,
'elapsed_time_seconds': 23.999904,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2025, 7, 7, 12, 46, 37, 62452),
'httpcompression/response_bytes': 187569,
'httpcompression/response_count': 9,
'httperror/response_ignored_count': 23,
'httperror/response_ignored_status_count/404': 23,
'log_count/ERROR': 9,
'log_count/INFO': 45,
'log_count/WARNING': 1,
'response_received_count': 32,
'scheduler/dequeued': 32,
'scheduler/dequeued/memory': 32,
'scheduler/enqueued': 32,
'scheduler/enqueued/memory': 32,
'spider_exceptions/KeyError': 9,
'start_time': datetime.datetime(2025, 7, 7, 12, 46, 13, 62548)}
2025-07-07 20:46:37 [scrapy.core.engine] INFO: Spider closed (finished)
最新发布
本文介绍了解决Scrapy框架中遇到的403错误的方法，通过在settings.py文件中设置HTTPERROR_ALLOWED_CODES参数为[403]来允许特定的状态码，确保爬虫能够继续运行。
扫一扫
6215
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
