Hadoop集群多次格式化导致容量分配为0不能导入数据

最新推荐文章于 2024-06-20 10:30:00 发布

boonya

最新推荐文章于 2024-06-20 10:30:00 发布

阅读量4.7k

点赞数 3

分类专栏： # Hadoop 大数据文章标签： hadoop fs put

本文链接：https://blog.youkuaiyun.com/boonya/article/details/85697815

版权

Hadoop 同时被 2 个专栏收录

31 篇文章

订阅专栏

大数据

14 篇文章

订阅专栏

遇到问题

使用hadoop fs -put 命令上传文件发现失败，报了以下错误：

03/01/19 15:18:03 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /home/input/file1.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
        at org.apache.hadoop.ipc.Client.call(Client.java:1347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy9.addBlock(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at $Proxy9.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)

上面的问题是由于没有分配有效的存储容量，hadoop dfsadmin -report 查看错误日志：

[root@master sbin]# hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: 0.00%
Replicated Blocks:
	Under replicated blocks: 0
	Blocks with corrupt replicas: 0
	Missing blocks: 0
	Missing blocks (with replication factor 1): 0
	Pending deletion blocks: 0
Erasure Coded Block Groups: 
	Low redundancy block groups: 0
	Block groups with corrupt internal blocks: 0
	Missing block groups: 0
	Pending deletion blocks: 0

解决问题

到sbin目录停止集群
```
./stop-all.sh
```

清理各个集群节点的数据目录（都需要清理否则还是不能分配）

rm -rf  /root/hadoop/tmp/*
rm -rf  /root/hadoop/var/*
rm -rf  /root/hadoop/dfs/name/*
rm -rf  /root/hadoop/dfs/data/*

到bin目录重新格式化
```
./hadoop namenode -format
```
重新启动集群
```
./start-all.sh
```

验证处理情况

[root@master sbin]# hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

Configured Capacity: 79401328640 (73.95 GB)
Present Capacity: 68053057536 (63.38 GB)
DFS Remaining: 68053049344 (63.38 GB)
DFS Used: 8192 (8 KB)
DFS Used%: 0.00%
Replicated Blocks:
	Under replicated blocks: 0
	Blocks with corrupt replicas: 0
	Missing blocks: 0
	Missing blocks (with replication factor 1): 0
	Pending deletion blocks: 0
Erasure Coded Block Groups: 
	Low redundancy block groups: 0
	Block groups with corrupt internal blocks: 0
	Missing block groups: 0
	Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (2):

Name: 192.168.1.11:9866 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 5674192896 (5.28 GB)
DFS Remaining: 34026467328 (31.69 GB)
DFS Used%: 0.00%
DFS Remaining%: 85.71%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Jan 03 02:14:20 EST 2019
Last Block Report: Thu Jan 03 02:13:55 EST 2019
Num of Blocks: 0


Name: 192.168.1.12:9866 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 5674078208 (5.28 GB)
DFS Remaining: 34026582016 (31.69 GB)
DFS Used%: 0.00%
DFS Remaining%: 85.71%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Jan 03 02:14:20 EST 2019
Last Block Report: Thu Jan 03 02:13:55 EST 2019
Num of Blocks: 0


[root@master sbin]#

可见已分配容量。