Dissecting The Nutch Crawler - Command "admin -create": net.nutch.tools.WebDBAdminTool

英文原文出处： DissectingTheNutchCrawler
转载本文请注明出处：http://blog.youkuaiyun.com/pwlazy

Command "admin -create": net.nutch.tools.WebDBAdminTool

> "admin: database administration, including creation"
> Usage: javanet.nutch.tools.WebDBAdminTooldb[-create][-textdumpdumpPrefix][-scoredump][-topk]

The "-create" options is a wrapper around "WebDBWriter.createWebDB(directory)". This in turn instantiates one WebDBWriter object with the arguments (dir, true) and then immediately calls ".close()" on the object.

Using "spam" as a directory name, let's run it and see what it creates:

$ bin/nutch admin spam -create
$ find spam -type file | xargs ls -l
-rw-r--r--  1 kangas  users   0 Oct 25 18:31 spam/dbreadlock
-rw-r--r--  1 kangas  users   0 Oct 25 18:31 spam/dbwritelock
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/linksByMD5/data
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/linksByMD5/index
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/linksByURL/data
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/linksByURL/index
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/pagesByMD5/data
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/pagesByMD5/index
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/pagesByURL/data
-rw-r--r--  1 kangas  users  16 Oct 25 18:31 spam/webdb/pagesByURL/index

---

命令”admin -create“ ： net.nutch.tools.WebDBAdminTool类

命令admin涉及数据库管理，包括产生新的数据库
调用方式
javanet.nutch.tools.WebDBAdminTooldb[-create][-textdumpdumpPrefix][-scoredump][-topk]

－create选项对应WebDBWriter.createWebDB(directory)方法，改方法首先用参数dir和ture实例化一个WebDBWriter实例然后立即调用该实例的close方法

我们现在用spam作为目录名，运行该命令看到底有什么发生

$bin / nutchadminspam - create
$findspam - typefile | xargsls - l
- rw - r -- r -- 1 kangasusers 0 Oct 25 18 : 31 spam / dbreadlock
- rw - r -- r -- 1 kangasusers 0 Oct 25 18 : 31 spam / dbwritelock
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / linksByMD5 / data
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / linksByMD5 / index
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / linksByURL / data
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / linksByURL / index
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / pagesByMD5 / data
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / pagesByMD5 / index
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / pagesByURL / data
- rw - r -- r -- 1 kangasusers 16 Oct 25 18 : 31 spam / webdb / pagesByURL / index