Datanode denied communication with namenode: DatanodeRegistration 解决办法

本文记录了在MacBook上配置Hadoop 2.2.0单机伪分布环境时遇到的问题及解决过程。主要问题是启动后未出现datanode进程,通过查看日志发现是由于namenode拒绝了datanode的通信请求。最终通过修改/etc/hosts文件指定了正确的主机名映射解决了该问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

安装环境: MacBook 10.9.1

Hadoop版本:2.2.0

单机伪同步环境。


启动start-dfs.sh后,发现本地没有datanode进程,查看datanode的日志如下:

2014-03-24 23:48:11,357 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564) service to localhost/192.168.1.101:9000 beginning handshake with NN

2014-03-24 23:48:11,381 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564) service to localhost/192.168.1.101:9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-2053126411-127.0.0.1-50010-1395699655564, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-ba95b66c-d94b-4390-8d44-adb486ba5682;nsid=2073699254;c=0)
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:739)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3929)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:948)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24079)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)


at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:695)
2014-03-24 23:48:11,382 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564) service to localhost/192.168.1.101:9000
2014-03-24 23:48:11,484 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564)
2014-03-24 23:48:11,484 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Removed bpid=BP-1316134947-127.0.0.1-1395699640023 from blockPoolScannerMap
2014-03-24 23:48:11,484 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-1316134947-127.0.0.1-1395699640023
2014-03-24 23:48:13,485 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-03-24 23:48:13,486 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2014-03-24 23:48:13,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1

************************************************************/

导致这个问题的原因很多,在stackoverflow上有很多解决方案,但都不适用我。经过摸索,通过修改/etc/hosts解决。

/etc/hosts原来为:

127.0.0.1  localhost

由于我本机IP地址为 192.168.1.101,所以将hosts文件修改为这个ip地址:

 192.168.1.101 localhost

理论上127.0.0.1也是可行的,但不知道我的为什么不行。


修改后,重新删除dfs.namenode.name.dir和dfs.datanode.data.dir以及hadoop.tmp.dir目录,分别执行:

hdfs namenode -format

start-dfs.sh

start-yarn.sh

后,用jps查看服务,全部正常:

37619 NodeManager

37798 Jps

37247 NameNode

37330 DataNode

37536 ResourceManager

37432 SecondaryNameNode


评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值