<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[No_oneelse的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/No_oneelse</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; No_oneelse]]></copyright><item><title><![CDATA[Scarpy源码分析18 总结]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744826</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744826</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 13:41:11 +0800</pubDate><description><![CDATA[2021SC@SDUSC

从10.01.021开始着手分析Scarpy，到现在两个多月的时间，前前后后写了17篇分析文章与技术报告。加总起来五万余字，收获颇丰。



在软件工程应用这门课中，我选择的是Scarpy爬虫源码的分析，虽然之前有过爬虫小项目的代码经验，但逻辑清晰的，系统全面的分析一个现有的成熟框架，对我来说是一个困难问题。特别是因自己身体各方面原因，10月一整个月都奔波在医院与学校中，整个月中工作与学习的时间屈指可数。在这样的情况下，用极为有限的时间，系统的分析整个成熟的框架，几乎成为了不可能]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析 17 Requests and Responses Ⅵ（终于肝完）]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744663</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744663</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 13:19:40 +0800</pubDate><description><![CDATA[2021SC@SDUSC

最后，作为软工应用这门课的收尾，熬夜硬肝，看完了response部分的代码。结合着官方文档进行分析：

Response objects：

类：scrapy.http.Response(*args,**kwargs)   源码附在最后

AResponseobject represents an HTTP response, which is usually downloaded (by the Downloader) and fed to the Spiders ...]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析 16 Requests and Responses Ⅴ]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744579</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744579</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 13:06:12 +0800</pubDate><description><![CDATA[类方法classmethodfrom_response(response[,formname=None,formid=None,formnumber=0,formdata=None,formxpath=None,formcss=None,clickdata=None,dont_click=False,...])




























]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析 15 Requests and Responses Ⅳ]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744466</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744466</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 12:59:25 +0800</pubDate><description><![CDATA[2021SC@SDUSC

PART5：Accessing additional data in errback functions在 errback 函数中访问附加数据


In case of a failure to process the request, you may be interested in accessing arguments to the callback functions so you can process further based on the arguments i.]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析 14 Requests and Responses Ⅲ]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744402</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744402</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 12:44:30 +0800</pubDate><description><![CDATA[2021SC@SDUSC

part3：将附加数据传递给回调函数


The callback of a request is a function that will be called when the response of that request is downloaded. The callback function will be called with the downloaded Response object as its first argument.


请求的回调函数是在下载请求响]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析 13 Requests and Responses Ⅱ]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744226</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744226</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 12:36:36 +0800</pubDate><description><![CDATA[2021SC@SDUSC

书接上回


	
	encoding(str) – the encoding of this request (defaults to'utf-8'). This encoding will be used to percent-encode the URL and to convert the body to bytes (if given as a string).
	
	
此请求的编码（默认为 'utf-8'）。此编码将用于对 URL 进行百分比编码并将正文转换为字节（..]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析 12 Requests and Responses]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121744105</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121744105</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 12:19:38 +0800</pubDate><description><![CDATA[2021SC@SDUSC

我计划在实验的最后时间中，对Requests and Responses部分进行重点的实验探究。重点理解源码中对于Requests and Responses的操作源码。

在项目文档中，有这样的定义：

Scrapy uses Request and Response objects for crawling web sites.

Typically, Request objects are generated in the spiders and pass across t]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析11 Item Pipeline Ⅱ]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121743979</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121743979</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 12:07:00 +0800</pubDate><description><![CDATA[2021SC@SDUSC

此外，pipeline还支持我们将项目写入 JSON 文件，我这里写一个简单的demo：


import json

from itemadapter import ItemAdapter

class JsonWriterPipeline:

    def open_spider(self, spider):
        self.file = open('items.jl', 'w')

    def close_spider(self, spider):
    ]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析10 Item Pipeline]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121743594</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121743594</guid><author>No_oneelse</author><pubDate>Mon, 06 Dec 2021 11:57:47 +0800</pubDate><description><![CDATA[2021SC@SDUSC

之前的1-9分析部分中，我们对scrapy源码的框架进行了总览分析，在了解了基本的scrapy爬虫项目结构后，我计划结合官方文档，对于scrapy爬虫中比较突出的几个部分进行重点代码分析。

首先是scrapy中Item Pipeline 的部分分析：

官方文档中对于scrapy是这么定义的：

Each item pipeline component (sometimes referred as just “Item Pipeline”) is a Python clas..]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析9]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121380446</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121380446</guid><author>No_oneelse</author><pubDate>Wed, 17 Nov 2021 15:55:19 +0800</pubDate><description><![CDATA[2021SC@SDUSC

ExecutionEngine 和 Scraper 关联的地方

跟了这么远，我们现在回到 ExecutionEngine 中的_next_request_from_scheduler方法，为了方便，我把这段代码再复制下放到这里来


# scrapy/core/engine.py

class ExecutionEngine(object):
    # ...

    def _next_request_from_scheduler(self, spider):
 ...]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析8]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121380293</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121380293</guid><author>No_oneelse</author><pubDate>Wed, 17 Nov 2021 15:52:55 +0800</pubDate><description><![CDATA[2021SC@SDUSC

具体_process_queue又做了什么呢


class Downloader(object):
    # ...

    def _process_queue(self, spider, slot):
        if slot.latercall and slot.latercall.active():
            return

        # Delay queue processing if a download_delay is con..]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析7]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121233285</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121233285</guid><author>No_oneelse</author><pubDate>Tue, 09 Nov 2021 17:44:25 +0800</pubDate><description><![CDATA[2021SC@SDUSC



3.4 Downloader

看下 fetch 方法


# scrapy/core/downloader/__init__.py

class Downloader(object):
    # ...

    def fetch(self, request, spider):
        def _deactivate(response):
            self.active.remove(request)
            return res]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析6]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121191462</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121191462</guid><author>No_oneelse</author><pubDate>Sun, 07 Nov 2021 14:51:21 +0800</pubDate><description><![CDATA[2021SC@SDUSC



回到 engine 的open_spider方法，这个时候 slot 相关的初始化以及启动，我们应该大致能理解了。后面依次启动了 scraper 和 crawler.stats，需要注意的是 downloader 是在 engine 刚创建的时候就已经初始化好了。另外，当这些都初始化好后，engine 发出了spider_opened信号。

前面我们提到过_next_request多次，我们顺着这个方法跟下去。


# scrapy/core/engine.py...]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析5]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121191447</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121191447</guid><author>No_oneelse</author><pubDate>Sun, 07 Nov 2021 14:49:54 +0800</pubDate><description><![CDATA[2021SC@SDUSC

3.3 ExecutionEngine

看下open_spider


# scrapy/core/engine.py

class ExecutionEngine(object):
    # ...

    @defer.inlineCallbacks
    def open_spider(self, spider, start_requests=(), close_if_idle=True):
        assert self.has_capacity(), .]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析4]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/121046630</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/121046630</guid><author>No_oneelse</author><pubDate>Sat, 30 Oct 2021 09:26:13 +0800</pubDate><description><![CDATA[2021SC@SDUSC



3.2 从入口开始分析

简单的认识了主要的几个类之后，我们来跟下代码。运行爬虫时，很多时候我们会这么执行


$ scrapy crawl &lt;spider_name&gt;


看下这个 scrapy 可执行文件


#!/Users/kevinbai/.pyenv/versions/3.7.5/envs/site_analysis/bin/python3.7
# -*- coding: utf-8 -*-
import re
import sys

from scr]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析3]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/120955073</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/120955073</guid><author>No_oneelse</author><pubDate>Mon, 25 Oct 2021 16:57:20 +0800</pubDate><description><![CDATA[2021SC@SDUSC

2 框架有哪些功能
要知道一个框架有些什么功能，可以看它的官方文档。

Scrapy 的主要部件有 Engine、Spider、Scheduler、Downloader、Item Pipelines 以及中间件。

更详细的功能，我们就可以通过主要部件去联想。比如：

Spider 涉及 Request、Response、Selector、Extractor
Scheduler 涉及去重的 DupeFilter
Downloader 涉及并发量、下载延时等设置
Item Pipe]]></description><category></category></item><item><title><![CDATA[Scarpy源码分析2]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/120954937</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/120954937</guid><author>No_oneelse</author><pubDate>Mon, 25 Oct 2021 16:37:22 +0800</pubDate><description><![CDATA[2021SC@SDUSC

DeferredList
当碰到需要等待多个 deferred 执行完毕的时候，我们可以使用 DeferredList

dl = defer.DeferredList([deferred1, deferred2, deferred3])
看个例子

from twisted.internet import defer

def print_result(result):
  for (success, value) in result:
    if success...]]></description><category></category></item><item><title><![CDATA[2021-10-05Scarpy 源码分析1]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/120615172</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/120615172</guid><author>No_oneelse</author><pubDate>Tue, 05 Oct 2021 16:26:37 +0800</pubDate><description><![CDATA[2021SC@SDUSC




1 Twisted

Scrapy 基于 Twisted，所以除了要有一定的 Python 基础外，还要对 Twisted 有一些了解。

我们可以将 Twisted 和 Asyncio 类比，它们都是为了支持协程而诞生的，只是前者比后者出现的更早。这 2 个技术实现的核心都是事件循环，当程序执行到某个耗时的 IO 操作时，程序的执行权限会被退回给事件循环，事件循环会检测其它准备就绪的协程，然后将执行权限交给它，当之前的协程 IO 操作完毕后，事件循环会将执行权限转给它，继]]></description><category></category></item><item><title><![CDATA[爬虫安装的3个问题与解决方案]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/120561393</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/120561393</guid><author>No_oneelse</author><pubDate>Thu, 30 Sep 2021 10:59:56 +0800</pubDate><description><![CDATA[2021SC@SDUSC

professor给的任务：选课同学每人每周要发布一篇技术博客，那么我就爬取同学们发布的博客。



计划采取的是java的gecco爬虫框架开发，但考虑到java语言对于爬虫的支持不如python的全面，以及在数据分析层面上python语言的充分支持，还是考虑采用python的scarpy爬虫对数据进行抓取。

安装scrapy时，（windows系统，python3.9）采用pip install scarpy 命令

报错1：10061由于目标计算机积极拒绝，无法连接

大]]></description><category></category></item><item><title><![CDATA[gecco爬虫初见与综述]]></title><link>https://blog.youkuaiyun.com/No_oneelse/article/details/120546695</link><guid>https://blog.youkuaiyun.com/No_oneelse/article/details/120546695</guid><author>No_oneelse</author><pubDate>Wed, 29 Sep 2021 13:30:49 +0800</pubDate><description><![CDATA[2021SC@SDUSC

Gecco是一款用java语言开发的轻量化的易用的网络爬虫。作为国内大佬研发的java网络爬虫，Gecco整合了jsoup、httpclient、fastjson、spring、htmlunit、redission等优秀框架，我们使用者只需要配置一些jquery风格的选择器就能很快的写出一个爬虫。同时这个Gecco框架有优秀的可扩展性，框架基于开闭原则进行设计，对修改关闭、对扩展开放。



以上是gecco爬虫官网的介绍，作为一个java爬虫，它

简单易用，使用jquery风]]></description><category></category></item></channel></rss>