There are 0 datanode(s) running and no node(s) are excluded in this operation

最新推荐文章于 2024-09-24 19:35:34 发布

转载最新推荐文章于 2024-09-24 19:35:34 发布 · 243 阅读

文章标签：

#大数据 #java

本文介绍了一个Hadoop环境中使用pi示例程序时遇到的错误及其解决方法。问题表现为无法复制文件到指定节点，并给出了详细的错误日志。通过删除特定文件夹后，问题得以解决。

1.在执行bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 5 10报错，

Number of Maps = 5
Samples per Map = 10
17/12/31 15:35:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/12/31 15:35:14 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/hadoop/QuasiMonteCarlo_1514705707054_1423939919/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1595)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3287)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:677)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:213)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:485)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)

2，百度了下，说是版本冲突。之后去/tmp/hadoop-hadoop/dfs/data下删除current文件夹，再次执行1的命令，就OK了
Virtual memory (bytes) snapshot=16252362752
                Total committed heap usage (bytes)=707153920
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=590
        File Output Format Counters
                Bytes Written=97
Job Finished in 249.718 seconds
Estimated value of Pi is 3.28000000000000000000