文章标题

最新推荐文章于 2022-10-17 11:19:04 发布

skyie

最新推荐文章于 2022-10-17 11:19:04 发布

阅读量394

点赞数

CC 4.0 BY-SA版权

分类专栏： python及其系列

本文链接：https://blog.youkuaiyun.com/skyie53101517/article/details/64147956

python及其系列专栏收录该内容

6 篇文章

订阅专栏

本文介绍使用Scrapy框架创建爬虫项目的步骤，包括定义Item、Spider及Pipeline等核心组件，并展示如何通过命令行启动爬虫抓取网页数据。

第一步

启动一个项目
定义自己的spider
定义自己的Itemline
定义Item pipeline存储数据

创建一个爬虫项目

scrapy startproject tutorial

生成如下文件

scrapy_test/
├── __init__.py
├── __init__.pyc
├── items.py
├── pipelines.py
├── settings.py
├── settings.pyc
└── spiders
    ├── dmoz_spider.py
    ├── dmoz_spider.pyc
    ├── __init__.py
    └── __init__.pyc

1 directory, 10 files

每个文件作用：

scrapy.cfg: 项目配置文件
tutorial/: 项目python模块, 呆会代码将从这里导入
tutorial/items.py: 项目items文件
tutorial/pipelines.py: 项目管道文件
tutorial/settings.py: 项目配置文件
tutorial/spiders: 放置spider的目录

定义item

from scrapy.item import Item, Field 
class DmozItem(Item):
    title = Field()
    link = Field()
    desc = Field()

定义自己的spider

from scrapy.spider import BaseSpider

class DmozSpider(BaseSpider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        open(filename, 'wb').write(response.body)

启动爬虫开始爬

scrapy crawl dmoz

看到启动的一些日志信息

2017-03-21 05:16:21 [scrapy] INFO: Scrapy 1.1.0dev1 started (bot: scrapy_test)
2017-03-21 05:16:21 [scrapy] INFO: Optional features available: ssl, http11
2017-03-21 05:16:21 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'scrapy_test.spiders', 'SPIDER_MODULES': ['scrapy_test.spiders'], 'BOT_NAME': 'scrapy_test'}
2017-03-21 05:16:22 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.memdebug.MemoryDebugger',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.spiderstate.SpiderState',
 'scrapy.extensions.throttle.AutoThrottle']
2017-03-21 05:16:23 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

定义自己的item来存储爬到的数据