spark一个版本问题引发的血案（java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler）

最新推荐文章于 2025-01-10 15:40:33 发布

原创最新推荐文章于 2025-01-10 15:40:33 发布 · 1.3w 阅读

7 ·

CC 4.0 BY-SA版权

Spark 专栏收录该内容

6 篇文章

订阅专栏

本文记录了一次因Spark版本不一致导致的java.lang.NoSuchMethodError异常问题，并分享了解决方案。作者发现项目中使用的Spark库版本与配置文件指定的版本不同，通过统一版本成功解决了异常。

先看异常：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jerry/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/data0/program/spark-1.6.1-bin-hadoop2.6.4/lib/spark-assembly-1.6.1-hadoop2.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/07/09 19:17:42 INFO SparkContext: Running Spark version 1.3.0
16/07/09 19:17:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/09 19:17:43 INFO SecurityManager: Changing view acls to: jerry
16/07/09 19:17:43 INFO SecurityManager: Changing modify acls to: jerry
16/07/09 19:17:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jerry); users with modify permissions: Set(jerry)
16/07/09 19:17:44 INFO Slf4jLogger: Slf4jLogger started
16/07/09 19:17:44 INFO Remoting: Starting remoting
16/07/09 19:17:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:4090]
16/07/09 19:17:44 INFO Utils: Successfully started service 'sparkDriver' on port 4090.
16/07/09 19:17:44 INFO SparkEnv: Registering MapOutputTracker
16/07/09 19:17:44 INFO SparkEnv: Registering BlockManagerMaster
16/07/09 19:17:44 INFO DiskBlockManager: Created local directory at /tmp/spark-a250cf38-ad2c-4f20-8cb2-22f33bd7c391/blockmgr-6a6dbba7-a1de-41fc-99d0-bf0856c81fcf
16/07/09 19:17:44 INFO MemoryStore: MemoryStore started with capacity 945.5 MB
16/07/09 19:17:44 INFO HttpFileServer: HTTP File server directory is /tmp/spark-ce6f474b-70a6-4edf-be1f-ab6e904f29c8/httpd-1ef8f1f4-4a16-4f59-b99a-6eeceb9ece31
16/07/09 19:17:44 INFO HttpServer: Starting HTTP Server
16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT
16/07/09 19:17:44 INFO AbstractConnector: Started SocketConnector@0.0.0.0:53269
16/07/09 19:17:44 INFO Utils: Successfully started service 'HTTP file server' on port 53269.
16/07/09 19:17:44 INFO SparkEnv: Registering OutputCommitCoordinator
16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT
16/07/09 19:17:44 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/07/09 19:17:44 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/07/09 19:17:44 INFO SparkUI: Started SparkUI at http://debian0:4040
16/07/09 19:17:44 INFO Executor: Starting executor ID <driver> on host localhost
16/07/09 19:17:44 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@localhost:4090/user/HeartbeatReceiver
16/07/09 19:17:45 INFO NettyBlockTransferService: Server created on 46819
16/07/09 19:17:45 INFO BlockManagerMaster: Trying to register BlockManager
16/07/09 19:17:45 INFO BlockManagerMasterActor: Registering block manager localhost:46819 with 945.5 MB RAM, BlockManagerId(<driver>, localhost, 46819)
16/07/09 19:17:45 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler(Ljava/lang/String;Ljava/lang/String;)V
	at org.apache.spark.sql.execution.ui.SQLTab.<init>(SQLTab.scala:34)
	at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)
	at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1369)
	at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)
	at com.spark.test.sql.SparkSqlJDBC2Mysql$.main(SparkSqlJDBC2Mysql.scala:21)
	at com.spark.test.sql.SparkSqlJDBC2Mysql.main(SparkSqlJDBC2Mysql.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

一开始想到的是scala版本的问题，所以就将scala版本由 2.10.6 修改成了 2.10.4,修改完以后，问题依旧。

然后就开始想，这是 spark 方法的问题，所以是spark版本的问题，由于自己导入了spark的lib库是1.6.1的，但是项目的pom文件中还是1.3.0版本的，所以修改成1.6.1以后，问题迎刃而解。

总结：注意版本版本版本的统一，重要的事情说三遍，一般scala遇到