Exception in createBlockOutputStream java.net.SocketException: 打开的文件过多

执行MapReduce程序时遇到错误,报错信息显示为'Exception in createBlockOutputStream java.net.SocketException: 打开的文件过多'。问题可能源于网络请求过多和未关闭的文件句柄。解决方案包括检查并适当地关闭文件流,以及优化处理网络请求的逻辑。

执行MapReduce程序报如下错误:

15/03/29 22:35:13 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.SocketException: 打开的文件过多
    at sun.nio.ch.Net.socket0(Native Method)
    at sun.nio.ch.Net.socket(Net.java:423)
    at sun.nio.ch.Net.socket(Net.java:416)
    at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:104)
    at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
    at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
    at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:62)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1608)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1281)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
15/03/29 22:35:13 WARN hdfs.DFSClient: DataStreamer Exception
java.lang.NullPointerException
    at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:321)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:643)
15/03/29 22:35:13 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1306947336-127.0.1.1-1427186980802:blk_1073741941_5216
java.lang.NullPointerException
    at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867)
15/03/29 22:35:13 INFO mapred.MapTask: Starting flush of map output
15/03/29 22:35:13 INFO mapred.MapTask: Spilling map output
15/03/29 22:35:13 INFO mapred.MapTask: bufstart = 0; bufend = 16070994; bufvoid = 104857600
15/03/29 22:35:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 15500404(62001616); length = 10713993/6553600
15/03/29 22:35:14 INFO mapred.LocalJobRunner: map > map
15/03/29 22:35:17 INFO mapred.LocalJobRunner: map > map
15/03/29 22:35:19 INFO mapred.MapTask: Finished spill 0
15/03/29 22:35:19 INFO mapred.LocalJobRunner: map task executor complete.
15/03/29 22:35:19 WARN mapred.LocalJobRunner: job_local592353110_0001
java.lang.Exception: java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1206)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
15/03/29 22:35:19 INFO mapreduce.Job: Job job_local592353110_0001 failed with state FAILED due to: NA
15/03/29 22:35:19 INFO mapreduce.Job: Counters: 25


原因:网络请求过多和打开的文件过多,没有释放一些文件句柄,程序打开了文件却没有在最后close。




评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值