DataNode启动失败问题解决

启动DataNode 提示Missing NameNode address

start all没有报错，但是发现这NameNode的webUI上面DataNode没有挂上。
进入DataNode查看日志发现下面问题。

• datanode 进程没有起来
• NodeManager启动过一段时间退出了。

错误 java.io.IOException: No services to connect, missing NameNode address.

2021-05-15 16:31:40,824 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unable to get NameNode addresses.
2021-05-15 16:31:40,907 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@33308786{/,null,UNAVAILABLE}{/datanode}
2021-05-15 16:31:40,921 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@4b6e2263{HTTP/1.1,[http/1.1]}{localhost:0}
2021-05-15 16:31:40,921 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@46b61c56{/static,file:///usr/local/hadoop/hadoop-3.2.1/share/hadoop/hdfs/webapps/static/,UNAVAILABLE}
2021-05-15 16:31:40,922 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@36060e{/logs,file:///usr/local/hadoop/hadoop-3.2.1/logs/,UNAVAILABLE}
2021-05-15 16:31:40,950 INFO org.apache.hadoop.ipc.Server: Stopping server on 9867
2021-05-15 16:31:40,951 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping DataNode metrics system…
2021-05-15 16:31:40,951 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system stopped.
2021-05-15 16:31:40,952 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system shutdown complete.
2021-05-15 16:31:40,963 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Shutdown complete.
2021-05-15 16:31:40,964 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: No services to connect, missing NameNode address.
at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.refreshNamenodes(BlockPoolManager.java:165)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1441)
at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:501)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2806)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2714)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2756)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2900)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2924)
2021-05-15 16:31:40,967 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.io.IOException: No services to connect, missing NameNode address.
2021-05-15 16:31:40,988 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
*

这个错误前面还有一个警告日志：
Unable to get NameNode addresses

NodeManager 过了一会也挂了

详细错误棧

2021-05-15 16:47:25,535 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 9 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:47:49,580 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Cache Size Before Cl
ean: 0, Total Deleted: 0, Public Deleted: 0, Private Deleted: 0
2021-05-15 16:47:56,541 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:47:57,542 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 1 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:47:58,546 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 2 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:47:59,547 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 3 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:00,549 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 4 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:01,552 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 5 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:02,556 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 6 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:03,558 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 7 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:04,559 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 8 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:05,563 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 9 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2021-05-15 16:48:05,564 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Unexpected error starting NodeStatusUpdater
java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:753)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)atorg.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy73.registerNodeManager(UnknownSource)atorg.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:73)atsun.ref$