Spark开发问题集锦1

19/06/04 11:12:00 WARN DFSClient: Slow ReadProcessor read fields took 41999ms (threshold=30000ms); ack: seqno: 38596 reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18446744031720466026 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK], DatanodeInfoWithStorage[10.66.89.25:50010,DS-11e29b01-afbb-4a9e-8c42-fb9916369fb7,DISK]]
19/06/04 11:12:00 WARN DFSClient: Slow waitForAckedSeqno took 41994ms (threshold=30000ms)
19/06/04 11:13:05 WARN DFSClient: Slow ReadProcessor read fields took 65401ms (threshold=30000ms); ack: seqno: 38597 reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18446744008319167405 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK], DatanodeInfoWithStorage[10.66.89.25:50010,DS-11e29b01-afbb-4a9e-8c42-fb9916369fb7,DISK]]
19/06/04 11:13:05 WARN DFSClient: Slow waitForAckedSeqno took 65397ms (threshold=30000ms)
19/06/04 11:13:05 WARN DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-755615917-10.31.147.97-1557914371710:blk_1077806083_4065325
java.io.IOException: Bad response ERROR for block BP-755615917-10.31.147.97-1557914371710:blk_1077806083_4065325 from datanode DatanodeInfoWithStorage[10.66.89.25:50010,DS-11e29b01-afbb-4a9e-8c42-fb9916369fb7,DISK]
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:781)
19/06/04 11:13:05 WARN DFSClient: Error Recovery for block BP-755615917-10.31.147.97-1557914371710:blk_1077806083_4065325 in pipeline DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK], DatanodeInfoWithStorage[10.66.89.25:50010,DS-11e29b01-afbb-4a9e-8c42-fb9916369fb7,DISK]: bad datanode DatanodeInfoWithStorage[10.66.89.25:50010,DS-11e29b01-afbb-4a9e-8c42-fb9916369fb7,DISK]
19/06/04 11:14:10 WARN DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-755615917-10.31.147.97-1557914371710:blk_1077806083_4173356
java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.31.147.16:39946 remote=emr-worker-2.cluster-115592/10.31.147.16:50010]
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:749)
19/06/04 11:14:10 WARN DFSClient: Error while syncing
java.io.IOException: All datanodes DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:891)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:417)
19/06/04 11:14:10 ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:891)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:417)
19/06/04 11:14:10 ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:891)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:417)
19/06/04 11:14:10 ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:891)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:417)
19/06/04 11:14:10 ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:891)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:417)
19/06/04 11:14:10 ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[10.31.147.16:50010,DS-95bf5309-dc91-45e4-af08-0420af685bcf,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:891)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:417)

具体原因不明；

解决方式：待解决。