spark application运行时版本不兼容错误

最新推荐文章于 2025-01-12 14:57:20 发布

xubc

最新推荐文章于 2025-01-12 14:57:20 发布

阅读量1.1k

点赞数

分类专栏： spark 文章标签： spark 版本兼容

本文链接：https://blog.youkuaiyun.com/u011098327/article/details/74231448

版权

spark 专栏收录该内容

14 篇文章

订阅专栏

17/06/27 14:34:41 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/06/27 14:34:41 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 788.8 KB, free 1246.5 MB)
17/06/27 14:34:41 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 54.0 KB, free 1246.4 MB)
17/06/27 14:34:41 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.50.70.121:37335 (size: 54.0 KB, free: 1247.2 MB)
17/06/27 14:34:41 INFO SparkContext: Created broadcast 0 from rdd at TradeInfoOutlier.scala:30
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
        at com.fangdd.data.profile.outlier.TradeInfoOutlier$.main(TradeInfoOutlier.scala:30)
        at com.fangdd.data.profile.outlier.TradeInfoOutlier.main(TradeInfoOutlier.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/06/27 14:34:42 INFO SparkContext: Invoking stop() from shutdown hook

这种错误是由于生产环境采用的是scala 2.10 + spark1.6.3的运行环境，本地打的application jar使用scala2.11 + spark.1.6.3的编译环境，所以放入生产环境集群报了上述错误，更改scala版本重新打jar包后运行成功