Scrapy自定义proxy代理及https证书

最新推荐文章于 2025-11-12 09:09:53 发布

原创最新推荐文章于 2025-11-12 09:09:53 发布 · 2.9k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#python #爬虫 #scrapy

python知识块同时被 2 个专栏收录

35 篇文章

订阅专栏

web

6 篇文章

订阅专栏

本文介绍如何在Scrapy中使用代理服务器进行抓取，包括通过环境变量和自定义下载中间件两种方式。同时，详细讲解了HTTPS自定义证书的配置方法，以应对不同网站的证书需求。

1.代理

方式一：使用默认
os.environ
{
http_proxy:http://root:woshiniba@192.168.11.11:9999/
https_proxy:http://192.168.11.11:9999/
}
方式二：使用自定义下载中间件

    def to_bytes(text, encoding=None, errors='strict'):
        if isinstance(text, bytes):
            return text
        if not isinstance(text, six.string_types):
            raise TypeError('to_bytes must receive a unicode, str or bytes '
                            'object, got %s' % type(text).__name__)
        if encoding is None:
            encoding = 'utf-8'
        return text.encode(encoding, errors)
        
    class ProxyMiddleware(object):
        def process_request(self, request, spider):
            PROXIES = [
                {'ip_port': '111.11.228.75:80', 'user_pass': ''},
                {'ip_port': '120.198.243.22:80', 'user_pass': ''},
                {'ip_port': '111.8.60.9:8123', 'user_pass': ''},
                {'ip_port': '101.71.27.120:80', 'user_pass': ''},
                {'ip_port': '122.96.59.104:80', 'user_pass': ''},
                {'ip_port': '122.224.249.122:8088', 'user_pass': ''},
            ]
            proxy = random.choice(PROXIES)
            if proxy['user_pass'] is not None:
                request.meta['proxy'] = to_bytes（"http://%s" % proxy['ip_port']）
                encoded_user_pass = base64.encodestring(to_bytes(proxy['user_pass']))
                request.headers['Proxy-Authorization'] = to_bytes('Basic ' + encoded_user_pass)
                print "**************ProxyMiddleware have pass************" + proxy['ip_port']
            else:
                print "**************ProxyMiddleware no pass************" + proxy['ip_port']
                request.meta['proxy'] = to_bytes("http://%s" % proxy['ip_port'])

# settings中设置    
DOWNLOADER_MIDDLEWARES = {
       'step8_king.middlewares.ProxyMiddleware': 500,
    }

2.https自定义证书

Https访问时有两种情况：

要爬取网站使用的可信任证书(默认支持)

DOWNLOADER_HTTPCLIENTFACTORY = “scrapy.core.downloader.webclient.ScrapyHTTPClientFactory”
DOWNLOADER_CLIENTCONTEXTFACTORY = “scrapy.core.downloader.contextfactory.ScrapyClientContextFactory”
要爬取网站使用的自定义证书

DOWNLOADER_HTTPCLIENTFACTORY = “scrapy.core.downloader.webclient.ScrapyHTTPClientFactory”
DOWNLOADER_CLIENTCONTEXTFACTORY = “step8_king.https.MySSLFactory”

# https.py
from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
from twisted.internet.ssl import (optionsForClientTLS, CertificateOptions, PrivateCertificate)

class MySSLFactory(ScrapyClientContextFactory):
    def getCertificateOptions(self):
        from OpenSSL import crypto
        v1 = crypto.load_privatekey(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.key.unsecure', mode='r').read())
        v2 = crypto.load_certificate(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.pem', mode='r').read())
        return CertificateOptions(
            privateKey=v1,  # pKey对象
            certificate=v2,  # X509对象
            verify=False,
            method=getattr(self, 'method', getattr(self, '_ssl_method', None))
        )
其他：
相关类
    scrapy.core.downloader.handlers.http.HttpDownloadHandler
    scrapy.core.downloader.webclient.ScrapyHTTPClientFactory
    scrapy.core.downloader.contextfactory.ScrapyClientContextFactory
相关配置
    DOWNLOADER_HTTPCLIENTFACTORY
    DOWNLOADER_CLIENTCONTEXTFACTORY