Python Crawler(2)Items and Pipelines

Python Crawler(2)Items and Pipelines

We can do Items.py as follow:
import scrapy


class QuoteItem(scrapy.Item):
# define the fields for your item here like: # name = scrapy.Field() text = scrapy.Field()
author = scrapy.Field()
tags = scrapy.Field()

class AuthorItem(scrapy.Item):
name = scrapy.Field()
desc = scrapy.Field()
birth = scrapy.Field()

We can do Pipelines.py as follow:

from scrapy.exceptions import DropItem

class DuplicatesPipeline(object):

def __init__(self):
self.names = set()

def process_item(self, item, spider):
name = item['name'] + ' - Unique' if name in self.names:
raise DropItem("Duplicate item found: %s" % item['name'])
else:
self.names.add(name)
item['name'] = name
return item

We can also as multiple pipelines
class MongoPipeline(object):

def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db

@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
)

def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]

def close_spider(self, spider):
self.client.close()

def process_item(self, item, spider):
collection_name = item.__class__.__name__
self.db[collection_name].insert(dict(item))
return item

ITEM_PIPELINES = {
'myproject.pipelines.PricePipeline': 300,
'myproject.pipelines.JsonWriterPipeline': 800,
}

Or in the settings.py
ITEM_PIPELINES = {
'tutorial.pipelines.DuplicatesPipeline': 300,
}

The pipeline actions will start from the low numbers.

Big sample Project
https://github.com/gnemoug/distribute_crawler

Deployment
https://scrapyd.readthedocs.io/en/latest/install.html
https://github.com/istresearch/scrapy-cluster

Install the scrapyd
>pip install scrapyd

Directly talk with Server side
https://scrapyd.readthedocs.io/en/latest/api.html

Clients
https://github.com/scrapy/scrapyd-client

Deploy
https://github.com/scrapy/scrapyd-client#scrapyd-deploy

Install the client
>pip install scrapyd-client

Start the Server
>scrapyd

Visit the console
http://localhost:6800/

Deploy my simple Things
>scrapyd-deploy
Packing version 1504042554
Deploying to project "tutorial" in http://localhost:6800/addversion.json
Server response (200):
{"status": "ok", "project": "tutorial", "version": "1504042554", "spiders": 2, "node_name": "ip-10-10-21-215.ec2.internal"}

List Target
>scrapyd-deploy -l
default http://localhost:6800/

Possible Cluster Solution in the future
https://github.com/istresearch/scrapy-cluster

Try with Python 3.6 later

References:
https://github.com/gnemoug/distribute_crawler
https://www.douban.com/group/topic/38361104/
http://wiki.jikexueyuan.com/project/scrapy/item-pipeline.html
https://segmentfault.com/a/1190000009229896