Crawler爬虫实例：huawei appstore

最新推荐文章于 2025-07-03 15:51:49 发布

原创最新推荐文章于 2025-07-03 15:51:49 发布 · 2.8k 阅读

0 ·

CC 4.0 BY-SA版权

Python 同时被 2 个专栏收录

13 篇文章

订阅专栏

Scrapy

2 篇文章

订阅专栏

本文介绍如何创建一个Scrapy项目，爬取华为AppStore的数据，并通过数据管道进行处理。首先，创建Scrapy项目'appstore'，接着定义要提取的数据模式，包括应用信息等字段。然后，编写爬虫'huawei_spider.py'，从华为AppStore获取数据。启用数据处理管道后，运行爬虫并查看爬取到的数据。最后，更新数据模式以添加新的字段，并重新运行爬虫。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1. create a scrapy project

>>> scrapy startproject appstore

2. define extracted data schema

edit appstore/appstore/items.py, add the following:

import scrapy


class AppstoreItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    url = scrapy.Field()
    appid = scrapy.Field()
    intro = scrapy.Field()

3. edit huawei_spider.py (example here: extract data from huawei appstore)

import scrapy
import re
from scrapy.selector import Selector
from appstore.items import AppstoreItem

class HuaweiSpider(scrapy.Spider):
    name = "huawei"
    allowed_domains = ["huawei.com"]

    start_urls = ["http://appstore.huawei.com/more/all"]

    def parse(self, response):
        page = Selector(response)

        divs = page.xpath('//div[@class="game-info  whole"]')

        for div in divs:
            item = AppstoreItem()
            item['title'] = div.xpath('.//h4[@class="title"]/a/text()'). \
                extract_first().encode('utf-8')
            item['url'] = div.xpath('.//h4[@class="title"]/a/@href').extract_first().encode('utf-8')
            appid = re.match(r'http://.*/(.*)', item['url']).group(1)
            item['appid'] = appid
            item['intro'] = div.xpath('.//p[@class="content"]/text()'). \
                extract_first().encode('utf-8')
            yield item

4. enable data pipeline in Scrapy

edit appstore/appstore/settings.py

ITEM_PIPELINES = {
   'appstore.pipelines.AppstorePipeline':300,
}
DOWNLOAD_DELAY=5

edit appstore/appstore/pipelines.py

class AppstorePipeline(object):
	def __init__(self):
		self.file = open('appstore.dat', 'wb')

	def process_item(self, item, spider):
	    	val = "{0}\t{1}\t{2}\n".format(item['appid'], item['title'], item['intro'])
		self.file.write(val)
		return item

5. run your spider

cd appstore

scrapy crawl huawei

cat appstore.dat

follow URLs in homepage

1. define extracted data schema -- add a new field to the schema

recommended = scrapy.Field()

2. modify huawei_spider.py

import scrapy
import re
from scrapy.selector import Selector
from appstore.items import AppstoreItem

class HuaweiSpider(scrapy.Spider):
    name = "huawei"
    allowed_domains = ["huawei.com"]

    start_urls = ["http://appstore.huawei.com/more/all"]

    def parse(self, response):
        page = Selector(response)

        hrefs = page.xpath('//h4[@class="title"]/a/@href')

        for href in hrefs:
            url = href.extract()
            yield scrapy.Request(url, callback = self.parse_item)

    def parse_item(self, response):
        page = Selector(response)
        item = AppstoreItem()

        item['title'] = page.xpath('//ul[@class="app-info-ul nofloat"]/li/p/span[@class="title"]/text()'). \
                extract_first().encode('utf-8')
        item['url'] = response.url
        appid = re.match(r'http://.*/(.*)', item['url']).group(1)
        item['appid'] = appid
        item['intro'] = page.xpath('//meta[@name="description"]/@content'). \
            extract_first().encode('utf-8')

        divs = page.xpath('//div[@class="open-info"]')
        recomm = ""

        for div in divs:
            url = div.xpath('./p[@class="name"]/a/@href').extract_first()
            recommended_appid = re.match(r'http://.*/(.*)', url).group(1)
            name = div.xpath('./p[@class="name"]/a/text()').extract_first().encode('utf-8')
            recomm += "{0}:{1},".format(recommended_appid, name)
        item['recommended'] = recomm
        yield item

3. run the spider

>>> scrapy crawl huawei

>>> cat appstore.dat