HDFS block丢失过多进入安全模式（safe mode）的解决方法

最新推荐文章于 2024-08-01 11:58:45 发布

原创

最新推荐文章于 2024-08-01 11:58:45 发布 · 2k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop

这篇博客记录了在HDFS中遇到的错误，包括NameNode处于安全模式和文件系统中存在损坏块的问题。通过执行`hdfs dfsadmin -safemode leave`退出安全模式，并使用`hdfs fsck -delete`命令删除损坏块以恢复文件系统的健康状态。

报错信息

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/user/root/.flink/application_1616479659685_0007/plugins/README.txt. Name node is in safe mode.
The reported blocks 5761 needs additional 14 blocks to reach the threshold 0.9990 of total blocks 5781.
The number of live datanodes 3 has reached the minimum number 1. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:devcdh2
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1439)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2372)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2318)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:771)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)

查看HDFS block丢失情况

 hdfs fsck /

Connecting to namenode via http://devcdh2:9870/fsck?ugi=root&delete=1&path=%2F
FSCK started by root (auth:SIMPLE) from /192.168.6.76 for path / at Tue Mar 23 16:02:31 CST 2021

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/83c4457d-249d-4bbd-a7c6-dcc74eb61b53: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448618

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/83c4457d-249d-4bbd-a7c6-dcc74eb61b53: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448620

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/a1ff8d8b-e377-4d4b-a13b-f7d464d7386c: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448619

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/a1ff8d8b-e377-4d4b-a13b-f7d464d7386c: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/4e6cc205-6f92-49a7-b0af-13d28a2a3f93: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448623

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/4e6cc205-6f92-49a7-b0af-13d28a2a3f93: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448624

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/bdf1ed78-9c1c-4beb-a517-ca9ac407b6d4: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767