03 python38的scrapy处理json格式数据

原创已于 2023-09-15 22:35:39 修改 · 535 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#scrapy #json

于 2023-09-15 22:34:25 首次发布

python38_scrapy 专栏收录该内容

4 篇文章

订阅专栏

1 json文档爬取或者获取ajax请求的json格式数据

1.0 网站分析

#网址：http://www.cwl.gov.cn/ygkj/wqkjgg/ssq/
#列表页分析：
第一页：http://www.cwl.gov.cn/cwl_admin/front/cwlkj/search/kjxx/findDrawNotice?name=ssq&issueCount=&issueStart=&issueEnd=&dayStart=&dayEnd=&pageNo=1&pageSize=30&week=&systemType=PC
第二页：http://www.cwl.gov.cn/cwl_admin/front/cwlkj/search/kjxx/findDrawNotice?name=ssq&issueCount=&issueStart=&issueEnd=&dayStart=&dayEnd=&pageNo=2&pageSize=30&week=&systemType=PC
最后一页：http://www.cwl.gov.cn/cwl_admin/front/cwlkj/search/kjxx/findDrawNotice?name=ssq&issueCount=&issueStart=&issueEnd=&dayStart=&dayEnd=&pageNo=542&pageSize=30&week=&systemType=PC

请求方式get
参数修改pageNo
返回数据为json

1.1 创建项目

scrapy startproject doubanSpider 
cd ssqSpider
scrapy genspider douban douban.com

1.2 创建爬虫

scrapy genspider douban "douban.com"

1.3 添加爬虫模型

import scrapy

class DoubanMovieItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field() #标题   
    stars = scrapy.Field() #分数   
    subject = scrapy.Field() #主题   
    pass

1.4 修改爬虫代码，列表页的多页数据为例子

1.5.1 数据为json文档时使用response.json()

import scrapy
from scrapy.http import HtmlResponse
from scrapy import Request
from ssqSpider.items import SsqspiderItem


class SsqSpider(scrapy.Spider):
    name = "ssq"
    allowed_domains = ["www.cwl.gov.cn"]
    start_urls = ["https://cwl.gov.cn"]
    # start_urls = ["http://www.cwl.gov.cn/cwl_admin/front/cwlkj/search/kjxx/findDrawNotice?name=ssq&issueCount=&issueStart=&issueEnd=&dayStart=&dayEnd=&pageNo=1&pageSize=30&week=&systemType=PC"]

    def start_requests(self):
        for page in range(1,55):   ## 1-54页，range为左闭右开
            url=f'http://www.cwl.gov.cn/cwl_admin/front/cwlkj/search/kjxx/findDrawNotice?name=ssq&issueCount=&issueStart=&issueEnd=&dayStart=&dayEnd=&pageNo={page}&pageSize=30&week=&systemType=PC'        
            yield Request(url=url,dont_filter=True)  # 如果有302跳转加入dont_filter=True

    def parse(self, response:HtmlResponse):       
        # print(response.json())       
        data=response.json()
        result=data['result']
        for i in result:
            item=SsqspiderItem()
            # print(i)
            item['qihao']=i['code']
            item['riqi']=i['date']
            item['kaijianghaoma_red']=i['red']
            item['kaijianghaoma_blue']=i['blue']
            
            item['jiangchijiner']=i['poolmoney']
            item['xiaoshouer']=i['sales']

            yield item

1.5.2 测试运行爬虫

scrapy crawl douban  #正式运行
或者
scrapy crawl douban -o douban.csv