Python Scrapy shell 调试添加headers

最新推荐文章于 2025-04-26 09:39:33 发布

原创最新推荐文章于 2025-04-26 09:39:33 发布 · 1.5k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#python #scrapy

博客展示了在terminal中使用Scrapy进行请求操作的过程。先输入scrapy shel，接着设置请求头，创建Scrapy请求对象并使用fetch方法发起请求，还给出了不同请求头设置下的操作及terminal显示的结果，涉及Scrapy模块和请求相关信息。

在terminal中输入

scrapy shel

随后再>>> 输入

headers= {‘host’:‘1’}
req = scrapy.Request(url=‘https://www.guazi.com/cd/buy/’,headers=headers)
fetch(req)

terminal具体显示结果如下：

[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x107fac748>
[s] item {}
[s] settings <scrapy.settings.Settings object at 0x107fac978>
[s] Useful shortcuts:
[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s] fetch(req) Fetch a scrapy.Request and update local objects
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser

headers = {‘user-agent’:’‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36’
',‘host’:‘1’
}
req = scrapy.Request(url=‘https://www.guazi.com/cd/buy/’,headers=headers)
fetch(req)
2019-03-19 14:11:31 [scrapy.core.engine] INFO: Spider opened
2019-03-19 14:11:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.guazi.com/cd/buy/> (referer: None)