Nutch基本命令

1：nutch读取hbase数据导出文本文件：./nutch readdb -dump /data/nutch_db/1108 -crawlId TestCrawl -content

会执行一个mr程序，/data/nutch_db/1108是mr的输出路径TestCrawl是hbase表名的前半部分。

2：

 inject         inject new urls into the database

 hostinject     creates or updates an existing host table from a text file

 generate       generate new batches to fetch from crawl db

 fetch          fetch URLs marked during generate

 parse          parse URLs marked during fetch

 updatedb       update web table after parsing

 updatehostdb   update host table after parsing

 readdb         read/dump records from page database

 readhostdb     display entries from the hostDB

 elasticindex   run the elasticsearch indexer

 solrindex      run the solr indexer on parsed batches

 solrdedup      remove duplicates from solr

 parsechecker   check the parser for a given url

 indexchecker   check the indexing filters for a given url

 plugin         load a plugin and run one of its classes main()

 nutchserver    run a (local) Nutch server on a user defined port

 junit          runs the given JUnit test

 or

 CLASSNAME      run the class named CLASSNAME

Most commands print help when invoked w/o parameters.

3：./crawl <seedDir> <crawlID> <solrURL> <numberOfRounds>

4，

    ./nutch inject /nutch/urls/seed.txt -crawlId 7day把url放入对应的hbase以bbs开头的bbs_webpage表中

    ./nutch generate -topN 5 -crawlId 7day

    ./nutch fetch -all -crawlId 7day -threads 20

    ./nutch parse -all -crawlId 7day

    ./nutch updatedb -all -crawlId 7day

    ./nutch index -D solr.server.url= http://192.168.4.129:8983/solr/ -all -crawlId 7day

    ./nutch  readdb -crawlId 7day_beijing -dump /home/nutch_output/beijing_1/

5， gora-hbase-mapping.xml该文件定义了列族及列的含义