Some Hints for Nutch

最新推荐文章于 2017-08-28 14:35:14 发布

Fenng

最新推荐文章于 2017-08-28 14:35:14 发布

阅读量166

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/fenng/article/details/81361802

本文分享了关于Nutch的几个小技巧，包括如何索引动态URL站点，可调整regex - urlfilter.txt或crawl - urlfilter.txt文件；编译Nutch要求Ant版本至少1.6以上；还介绍了验证regex - urlfilter是否正常的方法及示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

好久没关注Nutch了，看邮件列表，学到了几招关于 Nutch 的小技巧．

如何索引动态 URL 站点？

调整 regex-urlfilter.txt 或是 crawl-urlfilter.txt 文件．参见行”# skip URLs containing certain characters as probable queries,后面的内容．
编译 Nutch 需要用到的 Ant 版本至少要 1.6 以上．

验证regex-urlfilter是否正常(by Michael Nebel)：

If you want to know, if your regex-urlfilter works as expectet, you can
check it with the command:
cat FILE-WITH-URLS | nutch net/nutch/net/RegexURLFilter
or by calling "nutch net/nutch/net/RegexURLFilter" and entering the URL
by hand.
Everyline line beginning with a "+" ist accepted - a line with a "-" is
accepted. For example:
$ echo "http://www.nutch.org" | nutch net/nutch/net/RegexURLFilter
run with heapsize 256
-Xmx256m
050202 173520 loadingfile:/home/nutch/nutch-0.7/conf/nutch-default.xml
050202 173520 loading file:/home/nutch/nutch-0.7/conf/nutch-site.xml
050202 173520 found resource regex-urlfilter.txt at
file:/home/nutch/nutch-0.7/conf/regex-urlfilter.txt

Google+