scrapy Downloader Middlewares 中间件

最新推荐文章于 2025-06-03 09:12:43 发布

原创最新推荐文章于 2025-06-03 09:12:43 发布 · 445 阅读

0 ·

CC 4.0 BY-SA版权

scrapy 专栏收录该内容

9 篇文章

订阅专栏

本文介绍Scrapy框架中三种关键的中间件：UserAgentMiddleware用于随机更换User-Agent，CookiesMiddleware负责处理Cookie，而ProxyMiddleware则实现每次请求更换代理，有效避免IP被封，提升爬虫稳定性。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

class UserAgentMiddleware(object):
    """ Change User-Agent """

    def process_request(self, request, spider):
        agent = random.choice(agents)
        request.headers["User-Agent"] = agent


class CookiesMiddleware(object):
    """ Change Cookie """

    def process_request(self, request, spider):
        cookie = random.choice(cookies)
        request.cookies = cookie


class ProxyMiddleware(object):
    """Everyone Request change proxy"""

    def process_request(self, request, spider):
        try:
            #  get ip address
            PROXY_POOL_URL = 'xxxxxx'
            response = requests.get(url=PROXY_POOL_URL)
            if response.status_code == 200:
                proxy = response.text.strip()
                print('-------use---proxy------: http://',proxy)
                request.meta['proxy'] = "http://" + proxy

        except Exception as e :
            print("-----Error:---Proxy---ERROR-----", e)

url : https://scrapy-chs.readthedocs.io/zh_CN/0.24/topics/downloader-middleware.html