Scrapy报错

最新推荐文章于 2025-05-20 09:28:57 发布

Ian_Yan

最新推荐文章于 2025-05-20 09:28:57 发布

阅读量5.2k

点赞数 1

CC 4.0 BY-SA版权

文章标签： python scrapy AssertionError

本文链接：https://blog.youkuaiyun.com/Ian_Yan/article/details/78536034

本文介绍了一次使用Scrapy进行网页数据抓取时遇到的报错问题及其解决过程。错误发生在调用回调函数时，通过修改回调函数的引用方式解决了问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Scrapy抓取数据时报错

Traceback (most recent call last):
  File "C:\software\Python\Python35\lib\site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks
  result = g.send(result)
  File "C:\software\Python\Python35\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
  defer.returnValue((yield download_func(request=request,spider=spider)))
  File "C:\software\Python\Python35\lib\site-packages\twisted\internet\defer.py", line 1363, in returnValue
  raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 http://ios.jobbole.com/all-posts/page/2/>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\software\Python\Python35\lib\site-packages\scrapy\utils\defer.py", line 45, in mustbe_deferred
  result = f(*args, **kw)
  File "C:\software\Python\Python35\lib\site-packages\scrapy\core\spidermw.py", line 49, in process_spider_input
  return scrape_func(response, request, spider)
  File "C:\software\Python\Python35\lib\site-packages\scrapy\core\scraper.py", line 146, in call_spider
  dfd.addCallbacks(request.callback or spider.parse, request.errback)
  File "C:\software\Python\Python35\lib\site-packages\twisted\internet\defer.py", line 303, in addCallbacks
  assert callable(callback)
AssertionError

思考后，根据assert callable(callback)猜测是调用回调函数时发生了错误。检查源代码

  def parse(self, response):
      selector = Selector(response)
      # 获取文章的链接
      article_urls = selector.xpath('//a[@class="archive-title"]/@href').extract()
      for article_url in article_urls:
          yield Request(url=article_url, callback=self.parse_content)
      # 调用下一页的链接
      next_page_url = selector.xpath('//a[contains(@class, "next")]/@href').extract()
      if next_page_url:
          yield Request(url=next_page_url[0], callback="parse")#self.parse
      else:
          print("已经是最后一页了...........")

由于后面一个函数没有发挥作用，猜测这就是问题所在。所以将callback="parse"改为callback=self.parse后，问题解决