spark windows 下在idea中 作driver 调试 spark(spark本地代码集群运行问题)
18/03/15 19:35:48 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node1:38126] has failed, address is now gated for [5000] ms. Reason is: [org.apache.spark.TaskState$; local class incompatible: stream classdesc serialVersionUID = 746799155515967470, local class serialVersionUID = -2913614267616900700].
18/03/15 19:34:54 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@chen:2589] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
local class incompatible
Reason is: [Disassociated]
如果报错,应该仔细检查Worker Executor Driver Master的日志,看看是否有上面这样的内容,上面问题的原因是SCALA版本不一致造成的.
如何查看集群的SCALA版本
我是利用 spark-shell来查看的
Welcome to
__
/ / _ _/ /__
\ \/ \/ _ `/ _/ ‘/
// ./_,// //_\ version 1.3.1
/_/Using.Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)
Type in expressions to have them evaluated.
idea中的版本(我是maven导入的)
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_**2.10**</artifactId> <version>**1.3.1**</version> </dependency>
两个数字(2.10,1.3.0)都要注意
之后就可以在idea中,启动main函数充当driver来调度了
main中的要点
val sparkConf = new SparkConf().setAppName("Broadcast Test").
//集群的地址
setMaster("spark://node1:7077").
//先用Maven把IDEA中的项目打成jar包,然后填写这个Jar包的绝对路径,最后在idea中
//运行就好了
setJars(Seq("Z:\\sparkwokespace\\spark13\\target\\spark-1.0-SNAPSHOT.jar"))
val context: SparkContext = new SparkContext(sparkConf)