Unable to initialize any output collector(MapReduce执行到reduce过程失败抛出IO)

在执行MapReduce任务时遇到错误,报错信息显示为IO流错误。问题源于代码中输出部分引用的Text类不正确,导致无法初始化输出收集器。在main方法中,虽然map输出的key和value类型设置为org.apache.hadoop.io.Text,但在设置输出key和value类时引用的不是正确的Text类,从而引发异常。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

报错代码如下:

[hadoop@mini1 mypro]$ hadoop jar flow.jar  /flumcount/input  /flumcount/output

17/03/31 16:29:37 INFO client.RMProxy: Connecting to ResourceManager at mini1/192.168.40.201:8032
17/03/31 16:29:38 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/03/31 16:29:39 INFO input.FileInputFormat: Total input paths to process : 1
17/03/31 16:29:39 INFO mapreduce.JobSubmitter: number of splits:1
17/03/31 16:29:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1490947205776_0003
17/03/31 16:29:41 INFO impl.YarnClientImpl: Submitted application application_1490947205776_0003
17/03/31 16:29:41 INFO mapreduce.Job: The url to track the job: http://mini1:8088/proxy/application_1490947205776_0003/
17/03/31 16:29:41 INFO mapreduce.Job: Running job: job_1490947205776_0003
17/03/31 16:29:59 INFO mapreduce.Job: Job job_1490947205776_0003 running in uber mode : false
17/03/31 16:29:59 INFO mapreduce.Job:  map 0% reduce 0%
17/03/31 16:30:30 INFO mapreduce.Job: Task Id : attempt_1490947205776_0003_m_000000_0, Status : FAILED
Error: java.io.IOException: Unable to initialize any output collector  (无法初始化任何输出收集器
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)

        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

看错误信息报的是IO流错误,查看自己的代码:

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf);

job.setJarByClass(FlowClient.class);

job.setMapperClass(FLowMapper.class);
job.setReducerClass(FlowReducer.class);

job.setMapOutputKeyClass(Text.class);  // (这里下面的Text类应该都是org.apache.hadoop.io.Text)
job.setMapOutputValueClass(FlowBean.class);

job.setOutputKeyClass(Text.class);     //( 在输出这里引用的不是import org.apache.hadoop.io.Text;所以会出错)
job.setOutputValueClass(FlowBean.class);

FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

boolean rs = job.waitForCompletion(true);
System.exit(rs ? 0 : 1);
}

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值