conansonic的博客

nutch源码分析—inject本章开始分析nutch 1.12版本的源码，nutch在爬取网页时分为inject、generate、fetch、parse、updatedb五个步骤，本章先来看inject命令，nutch官网教程给出的实例如下， bin/nutch inject crawl/crawldb urls urls目录中的文件seed.txt包含了其实的url地址。 编译nutch

nutch源码分析—generate根据上一章的分析，“bin/nutch generate crawl/crawldb crawl/segments”这条命令最终会调用org.apache.nutch.crawl.Generator的main函数。Generator::main public static void main(String args[]) throws Except

nutch源码分析—solrindex“bin/nutch solrindex http://localhost:8983/solr crawl/crawldb/ -linkdb crawl/linkdb/ -dir crawl/segments/ -filter -normalize”elif [ "$COMMAND" = "solrindex" ] ; then CLASS="or

nutch源码分析—fetch

nutch源码分析—parsebin/nutch parse crawl/segments/*

nutch源码分析—updatedborg.apache.nutch.crawl.CrawlDb public static void main(String[] args) throws Exception { int res = ToolRunner.run(NutchConfiguration.create(), new CrawlDb(), args); Syst

nutch源码分析—invertlinksbin/nutch invertlinks crawl/linkdb -dir crawl/segments