scrapy自定义RetryMiddleware

最新推荐文章于 2025-02-28 13:17:04 发布

Karmenzind

最新推荐文章于 2025-02-28 13:17:04 发布

阅读量6.7k

点赞数 3

分类专栏：爬虫文章标签： python scrapy

本文链接：https://blog.youkuaiyun.com/sinat_22080643/article/details/75796276

版权

爬虫专栏收录该内容

1 篇文章

订阅专栏

本文介绍了如何在Scrapy项目中创建自定义RetryMiddleware，当JSON响应包含特定信息时，将请求加入重试队列。内容包括处理含有cookie禁用信息的策略，以及遵循Scrapy Download Middleware的process_response方法来调整请求行为，确保失败页面在爬虫完成所有常规页面抓取后重新安排下载。同时提到了在settings中激活该中间件以及重试时间和过滤规则的设置。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

爬虫repo地址：https://github.com/Karmenzind/EasyGoSpider

此处需求为：

返回json中带有{"code": 0}时，将此请求加入重试队列
假如json中含有cookie被禁信息，对cookie列表进行修正

源码注释中有一句：

Failed pages are collected on the scraping process and rescheduled at the end, once the spider has finished crawling all regular (non failed) pages.

继而根据Scrapy doc对通用Download Middleware中process_response的介绍：

If it returns a Request object, the middleware chain is halted and the returned request is rescheduled to be downloaded in the future. This is the same behavior as if a request is returned from process_request().

返回request对象时，该response不会再进入spider.parse_item方法，因而无需考虑在后者中的处理。

最后修改结果为：

class LocalRetryMiddleware(RetryMiddleware):

    def process_response(self, request, response, spider):
        if request.meta.get('dont_retry', False):
            return response

        if response.status in self.retry_http_codes:
            reason = response_status_message(response.status)
            return self._retry(request, reason, spider) or response

        # customiz' here
        resp_dct = json.loads(response.body)
        if resp_dct.get('code') != 0:
            reason = "Code is not 0."
            if resp_dct.get("data") == "\\u8be5\\u7528\\u6237\\u8bbf\\u95ee\\u6b21\\u6570\\u8fc7\\u591a".decode(
                    'unicode_escape'):  # 访问次数过多
                banned_cookie = response.request.cookies
                reason = "%s has been BANNED today." % banned_cookie
                spider.logger.warning(reason)
                spider.cookies.remove(banned_cookie)
                mongo_cli.cookies.find_one_and_update({"cookie": banned_cookie},
                                                      {"$set": {"FailedDate": str(datetime.date.today())}})
            return self._retry(request, reason, spider) or response

        return response

settings中激活

DOWNLOADER_MIDDLEWARES = {
    "scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware": None,
    "EasyGoSpider.middleware.LocalRetryMiddleWare": 302
    }

另外需要修改Retry time和Filter规则，此前已经设置，忽略。