本文介绍了一种增量网页模板检测框架，该方法能够即时处理爬取的网页，无需等待批量收集，减少了存储消耗并加速了数据刷新速度。实验表明，与传统方法相比，此框架将存储消耗降低了93%以上。
Yu Wang, Bingxing Fang, Xueqi Cheng, Li Guo, Hongbo Xu.
Incremental web page template detection.WWW '08: Proceeding of the 17th international conference on World Wide Web, ACM, 2008, 1247-1248.
Abstract: Most template detection methods process web pages in batches that a newly crawled page can not be processed until enough pages have been collected. This results in large storage consumption and a huge delay of data refreshing. In this paper, we present an incremental framework to detect templates in which a page is processed as soon as it has been crawled. In this framework, we don't need to cache any web page. Experiments show that our framework consumes less than 7% storage than traditional methods. And also the speed of data refreshing is accelerated because of the incremental manner.
Abstract: Most template detection methods process web pages in batches that a newly crawled page can not be processed until enough pages have been collected. This results in large storage consumption and a huge delay of data refreshing. In this paper, we present an incremental framework to detect templates in which a page is processed as soon as it has been crawled. In this framework, we don't need to cache any web page. Experiments show that our framework consumes less than 7% storage than traditional methods. And also the speed of data refreshing is accelerated because of the incremental manner.
扫一扫
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
