datanode Bad connect ack with firstBadLink

Hadoop Job启动慢及Datnode异常排查

最新推荐文章于 2021-10-12 11:42:56 发布

原创最新推荐文章于 2021-10-12 11:42:56 发布 · 1.6k 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

19 篇文章

订阅专栏

1、每次启动job很慢并有异常信息：

ERROR - java.io.IOException: Bad connect ack with firstBadLink as 10.21.232.114:50010
23-08-2016 14:13:21 CST import_ucord01_order_discount ERROR -   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1401)
23-08-2016 14:13:21 CST import_ucord01_order_discount ERROR -   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1302)
23-08-2016 14:13:21 CST import_ucord01_order_discount ERROR -   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
java.net.SocketTimeoutException: 70000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.21.193.201:60450 remote=/10.21.232.116:50010]
23-08-2016 14:18:09 CST import_ucord01_order_discount ERROR -   at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)

2、排除了防火墙和网络带宽的可能，登陆10.201.232.114查看日志频繁刷如下信息：

2016-08-23 14:13:04,862 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1905901049-10.21.232.111-1430129177585:blk_1310291686_237019930 src: /10.201.129.163:43292 dest: /10.201.232.114:50010
2016-08-23 14:13:04,862 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4786ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=5032ms
2016-08-23 14:13:04,863 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: cdh14.idc1.fn:50010:DataXceiver error processing unknown operation  src: /10.21.194.130:58540 dst: /10.21.232.114:50010
java.io.EOFException
        at java.io.DataInputStream.readShort(DataInputStream.java:315)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:56)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:202)
        at java.lang.Thread.run(Thread.java:745)

3、有日志看出，频繁的fullgc导致datanode进程停止服务