hadoop的datanode异常结束

最新推荐文章于 2024-05-14 00:25:41 发布

原创最新推荐文章于 2024-05-14 00:25:41 发布 · 1w 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

17 篇文章

订阅专栏

本文记录了一次Hadoop集群中datanode节点出现故障的排查过程，包括错误日志分析、网络及磁盘IO问题排查，并最终通过配置文件调整解决了问题。

集群datanode节点挂掉一个。错误如下:



2013-11-18 02:01:13,730 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.1.190:50010, storageID=DS-155659652-192.168.1.190-50010-1383619740465, infoPort=50075, ipcPort=50020):DataXceiver
java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 0 millis timeout left.
        at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:349)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.DataInputStream.read(DataInputStream.java:132)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:382)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
        at java.lang.Thread.run(Thread.java:619)
2013-11-18 02:01:14,731 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
2013-11-18 02:01:28,233 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_106200183580401997_8995735
2013-11-18 02:01:28,233 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Exiting DataBlockScanner thread.
2013-11-18 02:01:28,234 INFO org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: Shutting down all async disk service threads...
2013-11-18 02:01:28,234 INFO org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: All async disk service threads have been shut down.
2013-11-18 02:01:28,242 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2013-11-18 02:01:28,243 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at node-128-190/192.168.1.190
************************************************************/

到

(DataXceiver.java:

代码，相应代码如下:

      long startTime = DataNode.now();
      switch ( op ) {
      case DataTransferProtocol.OP_READ_BLOCK:
        readBlock( in );
        datanode.myMetrics.addReadBlockOp(DataNode.now() - startTime);
        if (local)
          datanode.myMetrics.incrReadsFromLocalClient();
        else
          datanode.myMetrics.incrReadsFromRemoteClient();
        break;
      case DataTransferProtocol.OP_WRITE_BLOCK:
        writeBlock( in );
        datanode.myMetrics.addWriteBlockOp(DataNode.now() - startTime);
        if (local)
          datanode.myMetrics.incrWritesFromLocalClient();

是网络io问题，但网卡没问题，联想到磁盘IO.把log向上找去，发现如下错误:

2013-11-18 02:01:11,027 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode.handleDiskError: Keep Running: false
2013-11-18 02:01:13,042 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down.
DataNode failed volumes:/data3/dfs/data/current;
2013-11-18 02:01:13,043 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-2665830487288
167569_12466770 java.io.IOException: Read-only file system
2013-11-18 02:01:13,043 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-2665830487288167569_12466770 1 : 
Thread is interrupted.
2013-11-18 02:01:13,043 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-2665830487288167569_1
2466770 terminating
2013-11-18 02:01:13,043 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-2665830487288167569_12466770 received 
exception java.io.IOException: Read-only file system
2013-11-18 02:01:13,043 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.1.190:50010, storageID=D
S-155659652-192.168.1.190-50010-1383619740465, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Read-only file system
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:260)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:480)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
        at java.lang.Thread.run(Thread.java:619)

未作修改，重启datanode,发现如下错误。印证了上面的想法:

2013-11-18 14:58:56,962 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-11-18 14:58:57,005 INFO org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started
2013-11-18 14:58:57,020 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-11-18 14:58:57,022 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-11-18 14:58:57,022 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2013-11-18 14:58:57,124 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-11-18 14:58:57,308 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: can not create directory: /data3/dfs/data
2013-11-18 14:58:57,829 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid value for volsFailed : 1 , Volumes tolerated : 0
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:974)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:403)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:309)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1651)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1590)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1608)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1734)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1751)

2013-11-18 14:58:57,831 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

在hdfs-site.xml里屏蔽掉坏磁盘。搞定。