scrapy+scrapyd+scrapydweb

scrapyd+scrapydweb [ubuntu]

可实现从scrapy的管理部署到监控定时运行等任务，scrapydweb也可以替换为Gerapy或Spiderkeeper。

install

pip3 install scrapyd
pip3 install scrapydweb

run

changebind_addressto0.0.0.0 in /usr/local/lib/python3.6/dist-packages/scrapyd/default_scrapyd.conf

bind_address = 0.0.0.0

run scrapyd

cd [your scrapyd server folder]
scrapyd

then scrapydweb

cd [your scrapydweb server folder]
scrapydweb

visit

for scrapydweb

127.0.0.1:5000

for scrapyd

127.0.0.1:6800

scrapy-client

install [ubuntu]

pip3 install scrapyd-client

scrapy demo:QuotesBot

可从QuotesBot获取

deploy

by scrapydclient

1. switch to QuotesBot project
2. append [deploy]in scrapy.cfg

[deploy]
url = http://[your scrapyd ip]:6800/
username = [your scrapyd username]
password = [your scrapyd password]
project = quotesbot

3. deploy

scrapyd-deploy

by scrapydweb

进入127.0.0.1:5000，在Deploy Project页面中上传爬虫压缩包并完成部署

run spider in scrapydweb

you can manage your scrapy project quotesbot

---

参考：
Scrapy
scrapyd
scrapydweb
scrapy-selenium