Hadoop序列化程序报错

最新推荐文章于 2023-11-12 23:33:28 发布

转载最新推荐文章于 2023-11-12 23:33:28 发布 · 443 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/ZouHongxue/p/7813023.html

文章标签：

#大数据 #java

本文解决了一个Hadoop MapReduce任务中出现的序列化问题，详细介绍了错误发生的原因在于序列化类的Read和Write操作不一致，以及如何通过确保使用基本类型并保持序列化顺序一致来修复该问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

这个错误指向序列化类，注意实现的Read和Wirte顺序一定要一致，并且要使用基本类型，不要使用包装类。

java.lang.Exception: java.io.EOFException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(Unknown Source)
at java.io.DataInputStream.readLong(Unknown Source)
at java.io.DataInputStream.readDouble(Unknown Source)
at com.demo.taobao.Product.readFields(Product.java:59)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146)
at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)