notes for pyspider

最新推荐文章于 2022-04-18 11:44:21 发布

原创最新推荐文章于 2022-04-18 11:44:21 发布 · 549 阅读

0 ·

CC 4.0 BY-SA版权

programming 专栏收录该内容

15 篇文章

订阅专栏

本文介绍了一种针对项目的任务数据库(TaskDB)设计方案，通过为每个项目创建独立的表来存储任务信息，包括任务ID、项目名、URL、状态等关键字段，并通过索引优化查询效率。文中还详细解释了schedule和fetch配置的具体含义。

taskdb

每个project 一张表，如

sqlite> .schema taskdb_test_meituan
CREATE TABLE `taskdb_test_meituan` (
                taskid PRIMARY KEY,
                project,
                url, status,
                schedule, fetch, process, track,
                lastcrawltime, updatetime
                );
CREATE INDEX `status_taskdb_test_meituan_index` ON `taskdb_test_meituan` (status);

字段意义

sqlite> select url, schedule, fetch from taskdb_test_meituan order by updatetime desc limit 2  ;
http://gj.meituan.com/category/jiafang/all/page1|{"priority": 2}|{"headers": {"Accept-Language": "zh-CN,zh;q=0.8", "Accept-Encoding": "gzip,deflate,sdch", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36", "Connection": "keep-alive", "Cache-Control": "max-age=0"}}
http://www.meituan.com/shop/2010830|{"priority": 2, "age": 172800}|{"headers": {"Accept-Language": "zh-CN,zh;q=0.8", "Accept-Encoding": "gzip,deflate,sdch", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36", "Connection": "keep-alive", "Cache-Control": "max-age=0"}}