https://scrapingclub.com/exercise/detail_cookie/

最新推荐文章于 2020-02-03 08:47:02 发布

转载最新推荐文章于 2020-02-03 08:47:02 发布 · 349 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/bamboozone/p/10443641.html

文章标签：

#python

本文详细介绍了如何使用Scrapy框架从响应头中解析并处理Cookie，包括提取token和设置自定义请求头，实现对特定网站的爬取。通过实际代码示例，读者可以学习到如何动态获取和应用Cookie，以解决网站的反爬虫策略。

    def parse(self, response):


        pattern=re.compile('token=(.*?);')
        token=pattern.findall( response.headers.get("set-cookie").decode("utf-8"))[0]
        cookie = {
            '__cfduid': 'd67f5270ed84c0000af9c771fdee950631551004073',
            '_ga': 'GA1.2.2009295084.1551004056',
            '_gid': 'GA1.2.513859849.1551004056',
            'token': token
        }
        yield scrapy.Request(url='https://scrapingclub.com/exercise/ajaxdetail_cookie/?token='+str(token),cookies=cookie,headers=header,callback=self.parse_json)

settings.py

COOKIES_ENABLED = True


DEFAULT_REQUEST_HEADERS = {
  # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  # 'Accept-Language': 'en',
# 'cookie': '__cfduid=d67f5270ed84c0000af9c771fdee950631551004073; _ga=GA1.2.2009295084.1551004056; _gid=GA1.2.513859849.1551004056; token=685NVSGV2O',
# 'dnt': '1',
'referer': 'https://scrapingclub.com/exercise/detail_cookie/',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36',
'x-requested-with': 'XMLHttpRequest'
}