Spark运行异常：java.lang.AbstractMethodError

最新推荐文章于 2023-08-05 11:10:32 发布

生命不息丶折腾不止

最新推荐文章于 2023-08-05 11:10:32 发布

阅读量5.9k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： spark 文章标签： spark 异常 exception

本文链接：https://blog.youkuaiyun.com/leen0304/article/details/78723536

spark 专栏收录该内容

58 篇文章

订阅专栏

本文分析了因编译和运行时使用不同版本Spark而引发的AbstractMethodError异常，指出此错误源于flatMap函数签名的变化，并提供了针对不同Spark版本的解决方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

异常日志：

17/12/05 18:30:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.AbstractMethodError: JavaKryoSerializer$1.call(Ljava/lang/Object;)Ljava/lang/Iterable;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
	at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
	at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
17/12/05 18:30:44 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,ANY, 2249 bytes)
17/12/05 18:30:44 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.AbstractMethodError: JavaKryoSerializer$1.call(Ljava/lang/Object;)Ljava/lang/Iterable;
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
    at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
...

异常分析：

这个错误是因为你编译代码时用的Spark 1.*，但是运行时用的Spark 2.*。flatMap的函数签名在这两个版本中是不一样的。

// 在Spark1.*中new FlatMapFunction 重写call方法返回类型为：Iterable
JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
//
    @Override
    public Iterable<String> call(String line) throws Exception {
            return Arrays.asList(line.split("\\\t"));
    }
});

// 在Spark2.*中new FlatMapFunction 重写call方法返回类型为：Iterable
JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
//
    @Override
    public Iterator<String> call(String line) throws Exception {
            return Arrays.asList(line.split("\\\t")).iterator();
    }
});