hadoop wrong key class: class org.apache.hadoop.io.Tex

最新推荐文章于 2022-12-18 14:25:07 发布

weixin_30776273

最新推荐文章于 2022-12-18 14:25:07 发布

阅读量509

点赞数 3

CC 4.0 BY-SA版权

文章标签：大数据 java

原文链接：http://www.cnblogs.com/csxf/p/3785728.html

本文解决了一个Hadoop MapReduce作业中出现的错误，即使用了不匹配的键类型导致的IOException。通过去除Combiner设置，解决了Reducer无法正确处理数据的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

java.io.IOException: wrong key class: class org.apache.hadoop.io.Text is not class com.sogou.web.selector.updana.wapPc.test.TextPair
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:172)
	at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1119)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1440)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:28)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:1)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:195)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1461)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1334)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1198)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608)
	at org.apache.had
java.io.IOException: wrong key class: class org.apache.hadoop.io.Text is not class com.sogou.web.selector.updana.wapPc.test.TextPair
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:172)
	at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1119)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1440)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:28)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:1)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:195)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1461)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1334)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1198)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608)
	at org.apache.had
java.io.IOException: wrong key class: class org.apache.hadoop.io.Text is not class com.sogou.web.selector.updana.wapPc.test.TextPair
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:172)
	at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1119)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1440)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:28)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:1)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:195)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1461)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1334)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1198)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608)
	at org.apache.had
java.io.IOException: wrong key class: class org.apache.hadoop.io.Text is not class com.sogou.web.selector.updana.wapPc.test.TextPair
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:172)
	at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1119)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1440)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:28)
	at com.sogou.web.selector.updana.wapPc.WapPcReducer.reduce(WapPcReducer.java:1)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:195)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1461)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1334)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1198)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608)
	at org.apache.had
修改方法：去掉combiner

//设置combiner
//job.setCombinerClass(Reducer.class);

去掉上述代码就行了，原因应该是设置的Reducer cannot be same due to inconsistency in the output key and value types.

转载于:https://www.cnblogs.com/csxf/p/3785728.html