scrapy中的request对象

Requtes与FormRequest详解

最新推荐文章于 2022-07-11 07:36:00 发布

转载最新推荐文章于 2022-07-11 07:36:00 发布 · 118 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/yugengde/p/8027524.html

文章标签：

#python

本文介绍了网络请求中两种常用的方法：Requtes 和 FormRequest。Requtes 支持 GET 和 POST 方法，允许设置 headers 和 cookies 等参数；FormRequest 专为 POST 请求设计，支持 formdata 参数自动转化为 URL 编码格式。

1. Requtes

注意 method = 'GET' headers = None 为字典类型 cookies = None 为字典类型

class Request(object_ref):

    def __init__(self, url, callback=None, method='GET', headers=None, body=None,
                 cookies=None, meta=None, encoding='utf-8', priority=0,
                 dont_filter=False, errback=None, flags=None):

        self._encoding = encoding  # this one has to be set first
        self.method = str(method).upper()
        self._set_url(url)
        self._set_body(body)
        assert isinstance(priority, int), "Request priority not an integer: %r" % priority
        self.priority = priority

        assert callback or not errback, "Cannot use errback without a callback"
        self.callback = callback
        self.errback = errback

        self.cookies = cookies or {}  
        self.headers = Headers(headers or {}, encoding=encoding)
        self.dont_filter = dont_filter

        self._meta = dict(meta) if meta else None
        self.flags = [] if flags is None else list(flags)

2. FormRequest

注意: method = 'POST' formdata 为字典类型

class FormRequest(Request):

    def __init__(self, *args, **kwargs):
        formdata = kwargs.pop('formdata', None)
        if formdata and kwargs.get('method') is None:
            kwargs['method'] = 'POST'

        super(FormRequest, self).__init__(*args, **kwargs)  # 指的就是request

        if formdata:
            items = formdata.items() if isinstance(formdata, dict) else formdata
            querystr = _urlencode(items, self.encoding)
            if self.method == 'POST':
                self.headers.setdefault(b'Content-Type', b'application/x-www-form-urlencoded')
                self._set_body(querystr)
            else:
                self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)