HDFS block丢失过多进入安全模式（safe mode）的解决方法

最新推荐文章于 2024-09-01 14:53:09 发布
原创 最新推荐文章于 2024-09-01 14:53:09 发布 · 2k 阅读 · 3 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#hadoop

这篇博客记录了在HDFS中遇到的错误，包括NameNode处于安全模式和文件系统中存在损坏块的问题。通过执行`hdfs dfsadmin -safemode leave`退出安全模式，并使用`hdfs fsck -delete`命令删除损坏块以恢复文件系统的健康状态。

 报错信息

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/user/root/.flink/application_1616479659685_0007/plugins/README.txt. Name node is in safe mode.
The reported blocks 5761 needs additional 14 blocks to reach the threshold 0.9990 of total blocks 5781.
The number of live datanodes 3 has reached the minimum number 1. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:devcdh2
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1439)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2372)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2318)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:771)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)

 查看HDFS block丢失情况

 hdfs fsck /

 

 

Connecting to namenode via http://devcdh2:9870/fsck?ugi=root&delete=1&path=%2F
FSCK started by root (auth:SIMPLE) from /192.168.6.76 for path / at Tue Mar 23 16:02:31 CST 2021

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/83c4457d-249d-4bbd-a7c6-dcc74eb61b53: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448618

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/83c4457d-249d-4bbd-a7c6-dcc74eb61b53: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448620

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/a1ff8d8b-e377-4d4b-a13b-f7d464d7386c: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448619

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/a1ff8d8b-e377-4d4b-a13b-f7d464d7386c: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/4e6cc205-6f92-49a7-b0af-13d28a2a3f93: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448623

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/4e6cc205-6f92-49a7-b0af-13d28a2a3f93: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448624

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/bdf1ed78-9c1c-4beb-a517-ca9ac407b6d4: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448622

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/bdf1ed78-9c1c-4beb-a517-ca9ac407b6d4: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/8bb95575-6366-4b33-bd75-5b27a503b94c: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448626

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/8bb95575-6366-4b33-bd75-5b27a503b94c: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448628

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/bc2a28b6-df8f-4d32-9ff9-635dd0687337: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448627

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/bc2a28b6-df8f-4d32-9ff9-635dd0687337: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/3c2b5cc6-4b14-480e-807a-2f8a1945bd79: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448630

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/3c2b5cc6-4b14-480e-807a-2f8a1945bd79: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/5efea232-a61a-4a9b-90bc-7b19fb1eebdc: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448631

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/5efea232-a61a-4a9b-90bc-7b19fb1eebdc: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448632

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/5eb9d4d3-62c6-4d06-9972-7bc6f0b19dbe: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448635

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/5eb9d4d3-62c6-4d06-9972-7bc6f0b19dbe: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/71bdf921-e9e8-443c-ad13-ce3e5f58c8de: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448634

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/71bdf921-e9e8-443c-ad13-ce3e5f58c8de: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448636

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/dbecccd1c9948bd10b9ed39062552d46/chk-16747/70d5887c-7987-4ef1-9317-b8993fa284e1:  Under replicated BP-199721927-192.168.6.76-1609304252767:blk_1073857082_116258. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/user/flink/checkpoint/dbecccd1c9948bd10b9ed39062552d46/chk-16747/_metadata:  Under replicated BP-199721927-192.168.6.76-1609304252767:blk_1073857084_116260. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/user/flink/checkpoint/dbecccd1c9948bd10b9ed39062552d46/chk-16747/eea2bc39-c441-4339-84cb-b5c994fe1ad2:  Under replicated BP-199721927-192.168.6.76-1609304252767:blk_1073857083_116259. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/user/flink/ha/completedCheckpoint09231ef9ff2b: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448633

/user/flink/ha/completedCheckpoint09231ef9ff2b: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpoint5e461d0cf506: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448621

/user/flink/ha/completedCheckpoint5e461d0cf506: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpoint740c86d9dede: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448625

/user/flink/ha/completedCheckpoint740c86d9dede: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpointd5d2a51448b6: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448637

/user/flink/ha/completedCheckpointd5d2a51448b6: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpointff0168a104e2: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448629

/user/flink/ha/completedCheckpointff0168a104e2: CORRUPT 1 blocks of total size 4505 B.
Status: CORRUPT
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    3883
 Total symlinks:                0

Replicated Blocks:
 Total size:    30817714125 B
 Total files:   5783
 Total blocks (validated):      5781 (avg. block size 5330862 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      20 (0.34596092 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        20
  CORRUPT BLOCKS:       20
  CORRUPT SIZE:         40955 B
  ********************************
 Minimally replicated blocks:   5761 (99.65404 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       3 (0.051894136 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.9885833
 Missing blocks:                0
 Corrupt blocks:                20
 Missing replicas:              6 (0.03459609 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Tue Mar 23 16:02:32 CST 2021 in 1798 milliseconds
FSCK ended at Tue Mar 23 16:02:32 CST 2021 in 1800 milliseconds
fsck encountered internal errors!


Fsck on path '/' FAILED
[root@devcdh1 data]# hadoop dfsadmin -safemode leave
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

Safe mode is OFF in devcdh1/192.168.6.76:8020
Safe mode is OFF in devcdh2/192.168.6.77:8020
[root@devcdh1 data]# hdfs fsck  /  -delete
Connecting to namenode via http://devcdh2:9870/fsck?ugi=root&delete=1&path=%2F
FSCK started by root (auth:SIMPLE) from /192.168.6.76 for path / at Tue Mar 23 16:04:25 CST 2021

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/83c4457d-249d-4bbd-a7c6-dcc74eb61b53: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448618

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/83c4457d-249d-4bbd-a7c6-dcc74eb61b53: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448620

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/a1ff8d8b-e377-4d4b-a13b-f7d464d7386c: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448619

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47124/a1ff8d8b-e377-4d4b-a13b-f7d464d7386c: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/4e6cc205-6f92-49a7-b0af-13d28a2a3f93: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448623

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/4e6cc205-6f92-49a7-b0af-13d28a2a3f93: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448624

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/bdf1ed78-9c1c-4beb-a517-ca9ac407b6d4: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448622

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47125/bdf1ed78-9c1c-4beb-a517-ca9ac407b6d4: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/8bb95575-6366-4b33-bd75-5b27a503b94c: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448626

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/8bb95575-6366-4b33-bd75-5b27a503b94c: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448628

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/bc2a28b6-df8f-4d32-9ff9-635dd0687337: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448627

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47126/bc2a28b6-df8f-4d32-9ff9-635dd0687337: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/3c2b5cc6-4b14-480e-807a-2f8a1945bd79: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448630

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/3c2b5cc6-4b14-480e-807a-2f8a1945bd79: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/5efea232-a61a-4a9b-90bc-7b19fb1eebdc: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448631

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/5efea232-a61a-4a9b-90bc-7b19fb1eebdc: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448632

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47127/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/5eb9d4d3-62c6-4d06-9972-7bc6f0b19dbe: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448635

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/5eb9d4d3-62c6-4d06-9972-7bc6f0b19dbe: CORRUPT 1 blocks of total size 1535 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/71bdf921-e9e8-443c-ad13-ce3e5f58c8de: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448634

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/71bdf921-e9e8-443c-ad13-ce3e5f58c8de: CORRUPT 1 blocks of total size 1240 B.
/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/_metadata: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448636

/user/flink/checkpoint/39e8593bbff53abc516aeeffeaf549bc/chk-47128/_metadata: CORRUPT 1 blocks of total size 911 B.
/user/flink/checkpoint/dbecccd1c9948bd10b9ed39062552d46/chk-16747/70d5887c-7987-4ef1-9317-b8993fa284e1:  Under replicated BP-199721927-192.168.6.76-1609304252767:blk_1073857082_116258. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/user/flink/checkpoint/dbecccd1c9948bd10b9ed39062552d46/chk-16747/_metadata:  Under replicated BP-199721927-192.168.6.76-1609304252767:blk_1073857084_116260. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/user/flink/checkpoint/dbecccd1c9948bd10b9ed39062552d46/chk-16747/eea2bc39-c441-4339-84cb-b5c994fe1ad2:  Under replicated BP-199721927-192.168.6.76-1609304252767:blk_1073857083_116259. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/user/flink/ha/completedCheckpoint09231ef9ff2b: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448633

/user/flink/ha/completedCheckpoint09231ef9ff2b: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpoint5e461d0cf506: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448621

/user/flink/ha/completedCheckpoint5e461d0cf506: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpoint740c86d9dede: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448625

/user/flink/ha/completedCheckpoint740c86d9dede: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpointd5d2a51448b6: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448637

/user/flink/ha/completedCheckpointd5d2a51448b6: CORRUPT 1 blocks of total size 4505 B.
/user/flink/ha/completedCheckpointff0168a104e2: CORRUPT blockpool BP-199721927-192.168.6.76-1609304252767 block blk_1075448629

/user/flink/ha/completedCheckpointff0168a104e2: CORRUPT 1 blocks of total size 4505 B.
Status: CORRUPT
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    3883
 Total symlinks:                0

Replicated Blocks:
 Total size:    30817714125 B
 Total files:   5783
 Total blocks (validated):      5781 (avg. block size 5330862 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      20 (0.34596092 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        20
  CORRUPT BLOCKS:       20
  CORRUPT SIZE:         40955 B
  ********************************
 Minimally replicated blocks:   5761 (99.65404 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       3 (0.051894136 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.9885833
 Missing blocks:                0
 Corrupt blocks:                20
 Missing replicas:              6 (0.03459609 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Tue Mar 23 16:04:26 CST 2021 in 763 milliseconds


The filesystem under path '/' is CORRUPT

 解决方案

 hadoop dfsadmin -safemode leave


// 注意，用delete会造成数据的丢失，

 hdfs fsck  /  -delete 
Status: HEALTHY
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    3883
 Total symlinks:                0

Replicated Blocks:
 Total size:    30817673170 B
 Total files:   5763
 Total blocks (validated):      5761 (avg. block size 5349361 B)
 Minimally replicated blocks:   5761 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       3 (0.05207429 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.9989586
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              6 (0.034716196 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Tue Mar 23 16:04:57 CST 2021 in 452 milliseconds


The filesystem under path '/' is HEALTHY

再次查看状态

hadoop fsck /

【Hadoop】HDFS block丢失过多进入安全模式（safe mode）的解决方法
九师兄
11-18 1453
hdfs 重启hdfs，发现HA集群总是nameNode 主备挂掉，然后报错原因导致dataNode丢失超过设置的丢失百分比，说明我们的损坏的文件比例超过了阈值, 这个阈值配置在hdfs中, 也就是说不允许任何一个块损坏掉. 如果我们配置成99%应该就不会触发safemode了.
一篇文章彻底理解 HDFS 的安全模式
明哥的IT随笔
11-19 3636
一篇文章彻底理解 HDFS 的安全模式
启动HDFS，由于block丢失过多进入安全模式（safe mode）
x²＋(y－√³x²)²＝1的博客
01-07 555
https://blog.youkuaiyun.com/wwg18895736195/article/details/84067412 https://www.pianshen.com/article/7626374810/
HDFS block丢失过多进入安全模式（safe mode）,不能在hdfs创建文件的解决方法
lesssssss的博客
04-20 1576
退出安全模式 hadoop dfsadmin -safemode leave。这个错误的原因是,丢失的block数量到达配置文件指定的阈值,hdfs开启了安全模式。删除丢失的block块 hdfs fsck / -delete。虚拟机任意目录下执行。
block missing导致hdfs进入安全模式的解决方式
manshan6084的博客
04-17 719
访问HDFS发现报错，出现提示安全模式开启，进入web端发现安全模式为开启。报告的区块121需要额外的6个区块才能达到总区块128的阈值0.9990。不要求活动datanode的最小数目。一旦达到阈值，安全模式将自动关闭。运行内存不足，存储空间不足，系统断电等其他原因导致datablock丢失。重启hadoop服务，重新进入HDFS web端查看发现安全模式已经关闭。再次检查hdfs文件信息，发现状态为HEALTHY。
HDFS的BLOCK损坏或丢失问题检查与处理
abyslll的博客
11-25 7130
1. fsck命令介绍 fsck是file system check的简写，中文名其实就是文件系统检查，通过hdfs fsck命令可以看出具体的参数。 [hdfs@rtn01 ~]$ hdfs fsck Usage: DFSck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-b...
HDFS block丢失，导致hadoop进入安全模式的解决方案
08-29
### HDFS Block丢失导致Hadoop进入安全模式的解决方案 在Hadoop分布式文件系统(HDFS)中，当系统检测到文件块(Block)丢失时，为了保护数据完整性并避免进一步的数据损失，Hadoop会自动进入安全模式(Safe Mode)。在...
Hdfs Name node is in safe mode
最新发布
技术博客
09-01 587
通俗来说就是，当前系统中元数据与记录数据块前后不一致，而 Hadoop 默认要求最大的误差阀值为 0.001，如果超过这个值，就会导致 Hadoop 一直处于安全模式。也就是说，我们其实已经实际删除了某些数据块，但是记录的元数据因为一些故障导致没有进行同步，就会出现这种情况。集群启动后，发现一直处于安全模式中，进入 WEB 界面查看，还有一大段提示，如。会发现那些数据块丢失的文件，将这些文件按照路径在 HDFS 中全部删除即可。解决方式：执行健康检查，删除损坏掉的block。然后删除不用的log文件即可。
记一次HDFS的block corrupt事件
weixin_34168880的博客
02-14 672
还有最后两天班，明天晚上回家过年了，可是CDH突然报了一个block missing的错误，用 hdfs fsck /检查了一下，我们的块一共有500W个，missing了将近100W个，天呐，不过由于Hdfs的replication的机制，只要不是3份全丢就可以修复，这样，绝大部分的块都修复了，但是还是有3000多个块是3份都丢失了，3份全丢后，状态就为corrupt，直接导致小时报和日报收到影...
HDFS Block损坏的解决方式与思考
呼呼的小窝
09-09 1671
背景&现象 断电导致HDFS服务不正常，并显示块损坏 恢复步骤 检查HDFS系统文件健康 $>hdfs fsck / 注：通过web ui也可以进行查看 检查是对应的哪些block发生了损坏 $>hdfs fsck -list-corruptfileblocks 出来的结果是损坏的block及对应的file所在的路径 生产场景分析 业务场景如下：MySQL ----同步数据--...
HDFS Block 损坏解决方案
Nice的博客
03-06 2069
背景描述： 机房断电重启后发现HDFS服务不正常 发现步骤： 检查HDFS文件系统健康 通过命令的方式查看或者web ui 信息进行查看 hdfs fsck / 检查对应哪些Block发生了损坏（显示具体的块信息和文件路径信息） hdfs fsck -list-corruptfileblocks 数据处理流程：MySQL-----> Hadoop，解决方式只需要重新同步一份该表的数据即可 深层次的思考： 如何获取文件块的具体信息？ 1个文件对应多个块，每个块分布在不同的机器上面？
Exception in thread “main“ java.lang.RuntimeException: org.apache.hadoop.hdfs.server.namenode.SafeMo
qq_48925150的博客
03-09 4611
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hdfs.server.namenode.SafeMo
分分钟解决HDFS无法启动的一个小问题------HDFS启动后一直处于安全状态
lin的专栏
03-04 5222
文章转自：原创：任航若泽大数据 问题 : HDFS启动后一直处于安全状态 解决过程： 1.查看hadoop namenode的启动日志 1STARTUP_MSG: Starting NameNode 2STARTUP_MSG: host = hadoop001/192.168.137.141 3STARTUP_MSG: args = [] 4STARTUP_MSG: v...
HDFS故障：Namenode安全模式：The reported blocks 12xx needs additional xx blocks to reach the threshold 0.999
peidezhi的专栏
04-15 7699
1 CDH环境，HDFS无法做任何操作，报错信息连接Namenode节点失败，处于safemode。 2 查看HDFS 实例，红色警告，不能创建/tmp/.cloudera..... 3 查看 namenode日志: /var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-cdh00.log.out 报错信息： org.apache.......
Hadoop启动后一直处于SafeMode
weixin_33696106的博客
03-04 908
2019独角兽企业重金招聘Python工程师标准>>> ...
Hadoop学习第四课（HDFS常用命令+Mapreduce-架构设计 ）
whs0329的博客
12-04 1251
1.HDFS命令与使用 hdfs dfs -Linux，-后面使用的命令和Linux的命令操作是一样的 1.1 Hadoop常用命令-----只放常用的 [hadoop@hadoop001 ~]$ hadoop Admin Commands: #admin命令 Client Commands: #客户端命令checknative check native Hadoop and compression libraries availab...
解决Hadoop Safe Mode的问题
Blossom
11-06 1328
Heron提交时出现如下问题： 解决： yitian@heron01:~/hadoop/hadoop-2.7.4$ bin/hadoop dfsadmin -safemode leave DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command fo...
hadoop fsck命令分析 + 源码解析
yukikaze_nanoda的博客
08-25 2019
fsck命令分析 HDFS支持fsck命令用以检查各种不一致。fsck用以报告各种文件问题，如block丢失或缺少block等。fack命令用法如下： $HADOOP_HOME/bin/hdfs fsck [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]] <path> 检查的起始目录 -move 将损坏的文件移动到/lost+found下面 -delete
生产HDFS Block损坏恢复最佳实践
lin的专栏
03-04 1544
1、上传文件hello.txt [root@cdh-node01 apps]# hdfs dfs -mkdir /blockrecover [root@cdh-node01 apps]# echo "hello word" &gt; hello.txt [root@cdh-node01 apps]# hdfs dfs -put hello.txt /blockrecover [root@c...
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值