Hadoop study notes - Datanode block scanner

最新推荐文章于 2024-03-30 12:29:15 发布

原创最新推荐文章于 2024-03-30 12:29:15 发布 · 129 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#大数据 #java

软件综合专栏收录该内容

39 篇文章

订阅专栏

本文深入探讨了Hadoop集群中DataBlockScanner的功能，解释了它如何周期性验证存储在DataNode上的块，以检测并修复潜在的坏块。提供了获取DataNode块验证报告的方法，并通过实例展示了报告内容，包括已验证块的数量、最近的验证状态等关键信息。

Every datanode runs a block scanner, which periodically verifies all the blocks stored
on the datanode. This allows bad blocks to be detected and fixed before they are read
by clients. The DataBlockScanner maintains a list of blocks to verify and scans them one
by one for checksum errors. The scanner employs a throttling mechanism to preserve
disk bandwidth on the datanode.

Blocks are periodically verified every three weeks to guard against disk errors over time
(this is controlled by the dfs.datanode.scan.period.hours property, which defaults to
504 hours). Corrupt blocks are reported to the namenode to be fixed.
You can get a block verification report for a datanode by visiting the datanode’s web
interface at [color=red][size=medium]http://datanode:50075/blockScannerReport[/size][/color]. Here’s an example of a report,
which should be self-explanatory:
Total Blocks : 194
Verified in last hour : 0
Verified in last day : 67
Verified in last week : 94
Verified in last four weeks : 187
Verified in SCAN_PERIOD : 187
Not yet verified : 7
Verified since restart : 70
Scans since restart : 1
Scan errors since restart : 0
Transient scan errors : 0
Current scan rate limit KBps : 1024
Progress this period : 0%
Time left in cur period : 99.47%

By specifying the listblocks parameter, [color=red][size=medium]http://datanode:50075/blockScannerReport?listblocks[/size][/color], the report is preceded by a list of all the blocks on the datanode along with
their latest verification status. Here is a snippet of the block list (lines are split to fit the
page):
blk_2880642477235589345_1712 : status : ok type : local scan time : 1328684200718 2012-02-08 01:56:40,718
blk_6862384560101574248_3203 : status : ok type : none scan time : 0 not yet verified
blk_-6204923618707049613_3146 : status : ok type : none scan time : 0 not yet verified
blk_8096385507793977436_1470 : status : ok type : local scan time : 1328726026095 2012-02-08 13:33:46,095
blk_-8383560827245098225_1470 : status : ok type : local scan time : 1328736508026 2012-02-08 16:28:28,026
blk_-1634356630613489001_3191 : status : ok type : none scan time : 0 not yet verified
blk_-6752468218406655007_3201 : status : ok type : none scan time : 0 not yet verified
blk_-1692843323764239407_1772 : status : ok type : local scan time : 1328742671906 2012-02-08 18:11:11,906
blk_3849616369028352463_3200 : status : ok type : none scan time : 0 not yet verified
blk_-34525423848470829_1226 : status : ok type : local scan time : 1328747018814 2012-02-08 19:23:38,814
blk_6305423182925037634_1226 : status : ok type : local scan time : 1328753126642 2012-02-08 21:05:26,642
blk_543251317099843969_3202 : status : ok type : none scan time : 0 not yet verified
blk_6417698981647840069_1833 : status : ok type : local scan time : 1328779874402 2012-02-09 04:31:14,402
blk_-7222269942471718886_3199 : status : ok type : none scan time : 0 not yet verified

check DataBlockScanner.java for detail