Scrapy从入门到放弃（一）

最新推荐文章于 2021-04-21 09:40:42 发布

weixin_30306905

最新推荐文章于 2021-04-21 09:40:42 发布

阅读量83

点赞数

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/Castbar/p/7407194.html

本文介绍使用Python及Scrapy框架爬取英雄联盟官方网站上的英雄信息，并将获取的数据保存到本地文件的过程。具体包括项目的搭建、定义数据项、编写爬虫逻辑及数据管道等关键步骤。

第一篇技术博客，现在开始一直不懈的追求技术的进步了hhh。

先从打好Python基础开始，利用Scrapy包爬取英雄联盟首页所有英雄的信息并保存到本地QAQ。

工具和环境

1.Python 2.7

2.IDE: Pycharm

3.Scrapy

代码实现

1.控制台

scrapy startproject lolheros
cd lolheros
scrapy genspider loler lol.qq.com

2.项目

#items
import scrapy


class LolherosItem(scrapy.Item):
    # define the fields for your item here like:
    names = scrapy.Field()

#spiders
import scrapy
from lolheros.items import LolherosItem


class LolerSpider(scrapy.Spider):
    name = 'loler'
    allowed_domains = ['lol.qq.com']
    start_urls = ['http://lol.qq.com/web201310/info-heros.shtml']

    def parse(self, response):
        items=[]
        item = LolherosItem()
        item['names'] = response.xpath('//*[@id="jSearchHeroDiv"]/li/a/text()').extract()
        items.append(item)
        return items

#pipelines
import os

class LolherosPipeline(object):
    def process_item(self, item, spider):
        base_dir = os.getcwd()
        filename = base_dir+'/lolheros.txt'
        with open(filename,'rb') as f:
            f.write(items)
        return items

#settings
BOT_NAME = 'lolheros'

SPIDER_MODULES = ['lolheros.spiders']
NEWSPIDER_MODULE = 'lolheros.spiders'

ROBOTSTXT_OBEY = True

ITEM_PIPELINES = {
    'lolheros.pipelines.LolherosPipeline': 300,
}

转载于:https://www.cnblogs.com/Castbar/p/7407194.html