Namenode实验二则

最新推荐文章于 2024-01-08 10:35:08 发布

jyjxs

最新推荐文章于 2024-01-08 10:35:08 发布

阅读量797

点赞数

分类专栏： Hadoop

本文链接：https://blog.youkuaiyun.com/jyjxs/article/details/19072021

版权

Hadoop 专栏收录该内容

3 篇文章

订阅专栏

本文详细介绍了如何在Hadoop集群中进行Namenode的格式化操作，解决namespaceID不匹配的问题，以及调整Namenode的检查点频率。在实验一中，通过停止集群、格式化Namenode、启动集群和检查datanode状态，解决了DataNode启动错误。在实验二中，通过修改`core-site.xml`配置文件将检查点周期从1小时缩短至2分钟，并验证了设置的有效性。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

实验一：格式化namenode

1、停止集群

[grid@node1 bin]$ stop-all.sh
stopping jobtracker
node3: stopping tasktracker
node2: stopping tasktracker
stopping namenode
node3: stopping datanode
node2: stopping datanode
node1: stopping secondarynamenode

2、格式化namenode

[grid@node1 conf]$ hadoop namenode -format
14/02/11 11:01:15 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node1/192.168.13.20
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.7.0_45
************************************************************/
Re-format filesystem in /home/grid/tmp/dfs/name ? (Y or N) Y
14/02/11 11:01:17 INFO util.GSet: Computing capacity for map BlocksMap
14/02/11 11:01:17 INFO util.GSet: VM type       = 32-bit
14/02/11 11:01:17 INFO util.GSet: 2.0% max memory = 932184064
14/02/11 11:01:17 INFO util.GSet: capacity      = 2^22 = 4194304 entries
14/02/11 11:01:17 INFO util.GSet: recommended=4194304, actual=4194304
14/02/11 11:01:18 INFO namenode.FSNamesystem: fsOwner=grid
14/02/11 11:01:18 INFO namenode.FSNamesystem: supergroup=supergroup
14/02/11 11:01:18 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/02/11 11:01:18 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/02/11 11:01:18 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/02/11 11:01:18 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/02/11 11:01:18 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/02/11 11:01:18 INFO common.Storage: Image file /home/grid/tmp/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds.
14/02/11 11:01:18 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/grid/tmp/dfs/name/current/edits
14/02/11 11:01:18 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/grid/tmp/dfs/name/current/edits
14/02/11 11:01:18 INFO common.Storage: Storage directory /home/grid/tmp/dfs/name has been successfully formatted.
14/02/11 11:01:18 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.13.20
************************************************************/

3、启动集群

[grid@node1 bin]$ start-all.sh
starting namenode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-namenode-node1.out
node3: starting datanode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-datanode-node3.out
node2: starting datanode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-datanode-node2.out
node1: starting secondarynamenode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-secondarynamenode-node1.out
starting jobtracker, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-jobtracker-node1.out
node3: starting tasktracker, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-tasktracker-node3.out
node2: starting tasktracker, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-tasktracker-node2.out

4、检查datanode

[grid@node2 ~]$ jps
4672 Jps
4601 TaskTracker

发现datanode节点中 DataNode进程没有出现

5、查看datanode的日志

[grid@node2 ~]$ cd /hadoop/hadoop-1.2.1/logs/

[grid@node2 logs]$ vim hadoop-grid-datanode-node2.log

.....

2014-02-11 11:07:55,181 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /home/grid/tmp/dfs/data:namenode namespaceID = 1323034225; datanode namespaceID = 700857896

....

发现如上启动错误

6、修改namenode的namespaceID

[root@node1 hadoop-1.2.1]# cd /home/grid/tmp/dfs/name/current

[root@node1 current]# cat VERSION
#Tue Feb 11 11:06:41 CST 2014
namespaceID=1323034225
cTime=0
storageType=NAME_NODE
layoutVersion=-41

[root@node1 current]# vim VERSION
[root@node1 current]# cat VERSION
#Tue Feb 11 11:06:41 CST 2014
namespaceID=700857896
cTime=0
storageType=NAME_NODE
layoutVersion=-41

[grid@node1 current]$ start-all.sh
starting namenode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-namenode-node1.out
node3: starting datanode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-datanode-node3.out
node2: starting datanode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-datanode-node2.out
node1: starting secondarynamenode, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-secondarynamenode-node1.out
starting jobtracker, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-jobtracker-node1.out
node3: starting tasktracker, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-tasktracker-node3.out
node2: starting tasktracker, logging to /hadoop/hadoop-1.2.1/libexec/../logs/hadoop-grid-tasktracker-node2.out

查看slave信息：

[grid@node2 current]$ jps
7651 Jps
7578 TaskTracker

查看master信息：

[grid@node1 current]$ cat VERSION
#Tue Feb 11 12:53:35 CST 2014
namespaceID=1323034225
cTime=0
storageType=NAME_NODE
layoutVersion=-41

发现namespaceID仍然没有被改变，不知道为什么，感觉修改namespaceID不可以吧，不知道别人怎么成功的。slave仍然报相同的错误。

7、修改datanode的spaceID

[grid@node2 current]$ cd /home/grid/tmp/dfs/data/current/

[grid@node2 current]$ cat VERSION
#Tue Feb 11 13:25:12 CST 2014
namespaceID=1323034225
storageID=DS-714125488-192.168.13.21-50010-1385203430467
cTime=0
storageType=DATA_NODE
layoutVersion=-41

8、重启集群验证:

[grid@node2 current]$ jps
7651 Jps
7578 TaskTracker
7456 DataNode

发现datanode 恢复正常。

实验二：修改namenode检查点频率

1、默认情况下namenode检查点为1小时

修改core-site.xml文件

添加如下：

<property>
<name>fs.checkpoint.period</name>
<value>120</value>
</property>

单位为秒。

2、检验试验结果

[grid@node1 current]$ date
Thu Feb 13 14:46:42 CST 2014
[grid@node1 current]$ ll
total 1044
-rw-r--r-- 1 grid oinstall 1048580 Feb 13 14:46 edits
-rw-r--r-- 1 grid oinstall     636 Feb 13 14:45 fsimage
-rw-r--r-- 1 grid oinstall       8 Feb 13 14:45 fstime
-rw-r--r-- 1 grid oinstall     101 Feb 13 14:45 VERSION

[grid@node1 current]$ date
Thu Feb 13 14:48:36 CST 2014
[grid@node1 current]$ ll
total 16
-rw-r--r-- 1 grid oinstall 4 Feb 13 14:47 edits
-rw-r--r-- 1 grid oinstall 636 Feb 13 14:47 fsimage
-rw-r--r-- 1 grid oinstall 8 Feb 13 14:47 fstime
-rw-r--r-- 1 grid oinstall 101 Feb 13 14:47 VERSION

2分钟检验一次，更新文件