Python Crawler(3)Services

最新推荐文章于 2025-12-05 17:02:52 发布

原创最新推荐文章于 2025-12-05 17:02:52 发布 · 124 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#python #json #ui

Summary 同时被 2 个专栏收录

381 篇文章

订阅专栏

Scripts

299 篇文章

订阅专栏

本文介绍如何使用Scrapyd服务启动、配置及监控Python爬虫项目。通过curl命令演示了如何启动爬虫、传递参数、列出项目和爬虫等操作，并提供了Scrapyd API和UI界面的访问方式。

Python Crawler(3)Services

Local Machine Service
Start the Service
>scrapyd

Call to start the services
>curl http://localhost:6800/schedule.json -d project=default -d spider=author
{"status": "ok", "jobid": "3b9c84c28dae11e79ba4a45e60e77f99", "node_name": "ip-10-10-21-215.ec2.internal"}

More API
http://scrapyd.readthedocs.io/en/stable/api.html#api

Call to Pass a Parameter
>curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d setting=DOWNLOAD_DELAY=2 -d arg1=val1

List Projects
>curl http://localhost:6800/listprojects.json
{"status": "ok", "projects": ["default", "tutorial"], "node_name": "ip-10-10-21-215.ec2.internal”}

List Spiders
>curl http://localhost:6800/listspiders.json?project=default
{"status": "ok", "spiders": ["author", "quotes"], "node_name": "ip-10-10-21-215.ec2.internal"}

UI of Status
http://localhost:6800/

http://scrapyd.readthedocs.io/en/stable/overview.html

Clustered Solution ?
https://github.com/rmax/scrapy-redis

References:
http://scrapyd.readthedocs.io/en/stable/overview.html#how-scrapyd-works