格式化namenode时报连接错误

最新推荐文章于 2021-08-11 11:43:12 发布

原创最新推荐文章于 2021-08-11 11:43:12 发布 · 6.2k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#namenode #格式化 #拒绝连接

hadoop 专栏收录该内容

30 篇文章

订阅专栏

本文记录了一次HDFS格式化失败的经历及解决过程。在HA集群环境下，由于JournalNode未启动导致NameNode无法完成格式化操作。通过调整配置并确保JournalNode正常运行后，问题得到解决。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

************************************************************/
18/09/05 21:18:00 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
18/09/05 21:18:00 INFO namenode.NameNode: createNameNode [-format]
18/09/05 21:18:01 WARN common.Util: Path /root/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
18/09/05 21:18:01 WARN common.Util: Path /root/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-7625d3f3-06cd-41bd-9ef6-6e880b19aebb
18/09/05 21:18:02 INFO namenode.FSNamesystem: No KeyProvider found.
18/09/05 21:18:02 INFO namenode.FSNamesystem: fsLock is fair: true
18/09/05 21:18:02 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
18/09/05 21:18:02 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
18/09/05 21:18:02 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
18/09/05 21:18:02 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
18/09/05 21:18:02 INFO blockmanagement.BlockManager: The block deletion will start around 2018 九月 05 21:18:02
18/09/05 21:18:02 INFO util.GSet: Computing capacity for map BlocksMap
18/09/05 21:18:02 INFO util.GSet: VM type = 64-bit
18/09/05 21:18:02 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
18/09/05 21:18:02 INFO util.GSet: capacity = 2^21 = 2097152 entries
18/09/05 21:18:02 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
18/09/05 21:18:02 INFO blockmanagement.BlockManager: defaultReplication = 3
18/09/05 21:18:02 INFO blockmanagement.BlockManager: maxReplication = 512
18/09/05 21:18:02 INFO blockmanagement.BlockManager: minReplication = 1
18/09/05 21:18:02 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
18/09/05 21:18:02 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
18/09/05 21:18:02 INFO blockmanagement.BlockManager: encryptDataTransfer = false
18/09/05 21:18:02 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
18/09/05 21:18:02 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
18/09/05 21:18:02 INFO namenode.FSNamesystem: supergroup = supergroup
18/09/05 21:18:02 INFO namenode.FSNamesystem: isPermissionEnabled = false
18/09/05 21:18:02 INFO namenode.FSNamesystem: Determined nameservice ID: ns1
18/09/05 21:18:02 INFO namenode.FSNamesystem: HA Enabled: true
18/09/05 21:18:02 INFO namenode.FSNamesystem: Append Enabled: true
18/09/05 21:18:02 INFO util.GSet: Computing capacity for map INodeMap
18/09/05 21:18:02 INFO util.GSet: VM type = 64-bit
18/09/05 21:18:02 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
18/09/05 21:18:02 INFO util.GSet: capacity = 2^20 = 1048576 entries
18/09/05 21:18:02 INFO namenode.FSDirectory: ACLs enabled? false
18/09/05 21:18:02 INFO namenode.FSDirectory: XAttrs enabled? true
18/09/05 21:18:02 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
18/09/05 21:18:02 INFO namenode.NameNode: Caching file names occuring more than 10 times
18/09/05 21:18:02 INFO util.GSet: Computing capacity for map cachedBlocks
18/09/05 21:18:02 INFO util.GSet: VM type = 64-bit
18/09/05 21:18:02 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
18/09/05 21:18:02 INFO util.GSet: capacity = 2^18 = 262144 entries
18/09/05 21:18:03 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
18/09/05 21:18:03 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
18/09/05 21:18:03 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
18/09/05 21:18:03 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
18/09/05 21:18:03 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
18/09/05 21:18:03 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
18/09/05 21:18:03 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
18/09/05 21:18:03 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
18/09/05 21:18:03 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/09/05 21:18:03 INFO util.GSet: VM type = 64-bit
18/09/05 21:18:03 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
18/09/05 21:18:03 INFO util.GSet: capacity = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /root/hadoop/dfs/name ? (Y or N) Y
18/09/05 21:18:07 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:07 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:08 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:08 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:09 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:09 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:10 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:10 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:11 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:11 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:12 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:12 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:13 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:13 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:14 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:14 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:15 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:15 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:16 INFO ipc.Client: Retrying connect to server: master/192.168.89.128:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:16 INFO ipc.Client: Retrying connect to server: master1/192.168.89.132:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
18/09/05 21:18:16 WARN namenode.NameNode: Encountered exception during format:
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if JNs are ready for formatting. 3 successful responses:
192.168.89.129:8485: false
192.168.89.131:8485: false
192.168.89.130:8485: false
1 exceptions thrown:
192.168.89.128:8485: Call From master/192.168.89.128 to master:8485 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
   at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
   at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
   at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232)
   at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:901)
   at org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:202)
   at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1011)
   at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1457)
   at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
18/09/05 21:18:16 ERROR namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if JNs are ready for formatting. 3 successful responses:
192.168.89.129:8485: false
192.168.89.131:8485: false
192.168.89.130:8485: false
1 exceptions thrown:
192.168.89.128:8485: Call From master/192.168.89.128 to master:8485 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
   at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
   at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
   at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232)
   at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:901)
   at org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:202)
   at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1011)
   at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1457)
   at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
18/09/05 21:18:16 INFO util.ExitUtil: Exiting with status 1
18/09/05 21:18:16 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.89.128
************************************************************/

原因分析：HA的集群必须依赖于JournalNode，如果NameNode无法获取JournalNode Quorum，HDFS则会无法格式化或无法启动。因此配置文件hdfs-site.xml至关重要，由于我在其中配置了namenode节点master为元数据存储在journalnode的路径，因此必须先启动master的journalnode线程。如果没有在master中配置，则在格式化namenode时不需要在master中启动journalnode。

解决办法：在namenode上启动journalnode线程后，再次尝试格式化namenode成功！