Scrapy之奇葩坑你爹:TypeError: 'Rule' object is not iterable

最新推荐文章于 2024-04-20 10:06:52 发布

原创最新推荐文章于 2024-04-20 10:06:52 发布 · 4.9k 阅读

5 ·

CC 4.0 BY-SA版权

Python 专栏收录该内容

17 篇文章

订阅专栏

本文详细解析了使用Scrapy爬虫时遇到的规则配置问题，特别关注于Rule对象的正确使用方式，通过修正逗号错误，解决了爬虫无法正常运行的严重问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

使用Scrapy 提取器

设置好规则

rules = (
    Rule(LinkExtractor(allow=r'type=4&page=\d+'), callback="parse_pages", follow=True)
)

。。。

其他代码省略，只说关键代码；

满心期待运行 scrapy crawl myspider_name

CRITICAL: Unhandled error in Deferred:

2018-07-24 18:52:17 [twisted] CRITICAL:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 79, in crawl
self.spider = self._create_spider(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 102, in _create_spider
return self.spidercls.from_crawler(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/crawl.py", line 100, in from_crawler
spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/__init__.py", line 51, in from_crawler
spider = cls(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/crawl.py", line 40, in __init__
self._compile_rules()
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/crawl.py", line 92, in _compile_rules
self._rules = [copy.copy(r) for r in self.rules]
TypeError: 'Rule' object is not iterable

尼玛，竟然报出一个严重的错误。

更为奇葩的是因为一个逗号的问题。。。

rules = (
    Rule(LinkExtractor(allow=r'type=4&page=\d+'), callback="parse_pages", follow=True),
)

在Rule规则之后加个逗号，重新运行scrapy 爬虫，错误解决。。。