HFDS 常用命令 fsck 查看文件信息和block位置信息

在HDFS中，提供了fsck命令，用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。

fsck命令必须由HDFS超级用户来执行，普通用户无权限。

[root@minit1 hadoop-2.7.2]# bin/hdfs fsck 
 Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [- locations | -racks]]]] [-includeSnapshots] [-            storagepolicies] [-blockId <blk_Id>]
        <path>  start checking from this path
        -move   move corrupted files to /lost+found
        -delete delete corrupted files
        -files  print out files being checked
        -openforwrite   print out files opened for write
        -includeSnapshots       include snapshot data if the given path indicates a snapshottable directory or there aresnapshottable directories under           it
        -list-corruptfileblocks print out list of missing blocks and files they belong to
        -blocks print out block report
        -locations      print out locations for every block
        -racks  print out network topology for data-node locations
        -storagepolicies        print out storage policy summary for the blocks
        -blockId        print out which file this blockId belongs to, locations (nodes, racks) of this block, andother diagnostics info(under replicated, corrupted or not, etc)

    查看文件中损坏的块（-list-corruptfileblocks）

   [root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -list-corruptfileblocks
Connecting to namenode via http://minit1:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
The filesystem under path '/user/mapreduce/wordcount/output5/part-r-00000' has 0 CORRUPT files

      

   将损坏的文件移动至/lost+found目录（-move）

[root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -move
Connecting to namenode via http://minit1:50070/fsck?ugi=root&move=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path /user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:00:05 CST 2017
.Status: HEALTHY
  Total size:    172 B
  Total dirs:    0
  Total files:   1
  Total symlinks:                0
  Total blocks (validated):      1 (avg. block size 172 B)
  Minimally replicated blocks:   1 (100.0 %)
  Over-replicated blocks:        0 (0.0 %)
  Under-replicated blocks:       0 (0.0 %)
  Mis-replicated blocks:         0 (0.0 %)
  Default replication factor:    3
  Average block replication:     3.0
 Corrupt blocks:                0
  Missing replicas:              0 (0.0 %)
  Number of data-nodes:          3
  Number of racks:               1
FSCK ended at Fri Dec 22 17:00:05 CST 2017 in 1 milliseconds

 删除损坏的文件（-delete）

[root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -delete
Connecting to namenode via http://minit1:50070/fsck?ugi=root&delete=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path/user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:03:59 CST 2017
.Status: HEALTHY
   Total size:    172 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 172 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
  Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Dec 22 17:03:59 CST 2017 in 0 milliseconds

       

   检查并列出所有文件状态（-files）

 [root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -files
Connecting to namenode via http://minit1:50070/fsck?ugi=root&files=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path /user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:05:47 CST 2017
/user/mapreduce/wordcount/output5/part-r-00000 172 bytes, 1 block(s):  OK
Status: HEALTHY
 Total size:    172 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 172 B)
  Minimally replicated blocks:   1 (100.0 %)
  Over-replicated blocks:        0 (0.0 %)
  Under-replicated blocks:       0 (0.0 %)
  Mis-replicated blocks:         0 (0.0 %)
  Default replication factor:    3
  Average block replication:     3.0
  Corrupt blocks:                0
  Missing replicas:              0 (0.0 %)
  Number of data-nodes:          3
  Number of racks:               1
FSCK ended at Fri Dec 22 17:05:47 CST 2017 in 1 milliseconds

   检查并打印正在被打开执行写操作的文件（-openforwrite）

[root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -openforwrite
Connecting to namenode via http://minit1:50070/fsck?ugi=root&openforwrite=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path /user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:08:21 CST 2017
.Status: HEALTHY
 Total size:    172 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 172 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Dec 22 17:08:21 CST 2017 in 0 milliseconds

 

打印文件的Block报告（-blocks）

   [root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -files -blocks
Connecting to namenode via http://minit1:50070/fsck?ugi=root&files=1&blocks=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path /user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:09:33 CST 2017
/user/mapreduce/wordcount/output5/part-r-00000 172 bytes, 1 block(s):  OK
0. BP-1296710928-192.168.65.145-1513844600367:blk_1073741891_1067 len=172 repl=3


Status: HEALTHY
 Total size:    172 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 172 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Dec 22 17:09:33 CST 2017 in 0 milliseconds

打印文件块的位置信息（-locations）

[root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -files -blocks -locations
Connecting to namenode via http://minit1:50070/fsck?ugi=root&files=1&blocks=1&locations=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path /user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:10:33 CST 2017
/user/mapreduce/wordcount/output5/part-r-00000 172 bytes, 1 block(s):  OK
0. BP-1296710928-192.168.65.145-1513844600367:blk_1073741891_1067 len=172 repl=3 [DatanodeInfoWithStorage[192.168.65.146:50010,DS-0838f58c-6908-4651-a027-e36acc3d5cf8,DISK], DatanodeInfoWithStorage[192.168.65.145:50010,DS-658fbbe4-88c7-4815-a09d-5d69bb63dfe8,DISK], DatanodeInfoWithStorage[192.168.65.147:50010,DS-6b378552-8cac-4312-bfb2-858755402788,DISK]]


Status: HEALTHY
 Total size:    172 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 172 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Dec 22 17:10:33 CST 2017 in 1 milliseconds

打印文件块位置所在的机架信息（-racks）

[root@minit1 hadoop-2.7.2]# bin/hdfs fsck  /user/mapreduce/wordcount/output5/part-r-00000 -files -blocks -locations -racks
Connecting to namenode via http://minit1:50070/fsck?ugi=root&files=1&blocks=1&locations=1&racks=1&path=%2Fuser%2Fmapreduce%2Fwordcount%2Foutput5%2Fpart-r-00000
FSCK started by root (auth:SIMPLE) from /192.168.65.145 for path /user/mapreduce/wordcount/output5/part-r-00000 at Fri Dec 22 17:12:27 CST 2017
/user/mapreduce/wordcount/output5/part-r-00000 172 bytes, 1 block(s):  OK
0. BP-1296710928-192.168.65.145-1513844600367:blk_1073741891_1067 len=172 repl=3 [/default-rack/192.168.65.146:50010, /default-rack/192.168.65.145:50010, /default-rack/192.168.65.147:50010]


Status: HEALTHY
 Total size:    172 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 172 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Dec 22 17:12:27 CST 2017 in 1 milliseconds