scrapy 爬取当当网产品分类

最新推荐文章于 2023-12-23 22:39:46 发布

转载最新推荐文章于 2023-12-23 22:39:46 发布 · 78 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/Erick-L/p/6835391.html

文章标签：

#python #json

本文介绍了一个使用Scrapy框架实现的爬虫项目，该项目针对当当网的汽车类图书进行数据抓取，包括商品名称、价格、链接及评价数量等信息，并详细展示了spider部分、pipeline部分和item部分的具体代码实现。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

#spider部分
import scrapy
from Autopjt.items import AutopjtItem
from scrapy.http import Request

class AutospdSpider(scrapy.Spider):
    name = "autospd"
    allowed_domains = ["dangdang.com"]
    start_urls = ['http://category.dangdang.com/pg1-cid4007379.html']

    def parse(self, response):
        item = AutopjtItem()
        item['name'] =response.xpath('//a[@name="itemlist-title"]/@title').extract()
        item['price'] = response.xpath('//span[@class="price_n"]/text()').extract()
        item['link'] = response.xpath('//a[@name="itemlist-title"]/@href').extract()
        item['comnum'] = response.xpath('//a[@name="itemlist-review"]/text()').extract()
        yield item
        for i in range(1,101):
            url = 'http://category.dangdang.com/pg'+str(i)+'-cid4007379.html'
            yield Request(url,callback=self.parse)

pipeline部分

import codecs
import json

class AutopjtPipeline(object):
    def __init__(self):
        self.file = codecs.open('D:/mydata.json','wb',encoding='utf-8')
    def process_item(self, item, spider):
        for j in range(0,len(item['name'])):
            name = item['name'][j]
            price = item['price'][j]
            comnum = item['comnum'][j]
            link =item['link'][j]
            goods = {'name':name,'price':price,'comnum':comnum,'link':link}
            i = json.dumps(dict(goods),ensure_ascii=False)
            line = i + '\n'
            self.file.write(line)
        return item
    def close_spider(self,spider):
        self.file.close()

item部分

import scrapy


class AutopjtItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    name = scrapy.Field()
    price = scrapy.Field()
    link = scrapy.Field()
    comnum = scrapy.Field()

转载于:https://www.cnblogs.com/Erick-L/p/6835391.html