Spark开发环境搭建

开发环境

Spark 2.4.4使用Scala2.12:

Java安装

bash-3.2$ java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

scala安装

解压

tar -zxvf scala-2.12.10.tgz

配置

vi .bash_profile
export SCALA_HOME=/Users/linhongzheng/software/scala-2.12.10
export PATH=$PATH:$SCALA_HOME/bin

生效

source .bash_profile

验证

bash-3.2$ scala
Welcome to Scala 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.2).
Type in expressions for evaluation. Or try :help.

scala> 1+2
res0: Int = 3

Spark安装

解压

tar -zxvf spark-2.4.4-bin-hadoop2.7.tgz

配置

vi .bash_profile
export SPARK_HOME=/Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH

生效

source .bash_profile

验证

运行官网的例子

bash-3.2$ $SPARK_HOME/bin/run-example SparkPi 10
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7/jars/spark-unsafe_2.11-2.4.4.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
19/11/11 09:53:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/11/11 09:53:25 INFO SparkContext: Running Spark version 2.4.4
19/11/11 09:53:25 INFO SparkContext: Submitted application: Spark Pi
19/11/11 09:53:25 INFO SecurityManager: Changing view acls to: linhongzheng
19/11/11 09:53:25 INFO SecurityManager: Changing modify acls to: linhongzheng
19/11/11 09:53:25 INFO SecurityManager: Changing view acls groups to:
19/11/11 09:53:25 INFO SecurityManager: Changing modify acls groups to:
19/11/11 09:53:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(linhongzheng); groups with view permissions: Set(); users  with modify permissions: Set(linhongzheng); groups with modify permissions: Set()
19/11/11 09:53:25 INFO Utils: Successfully started service 'sparkDriver' on port 58107.
19/11/11 09:53:25 INFO SparkEnv: Registering MapOutputTracker
19/11/11 09:53:25 INFO SparkEnv: Registering BlockManagerMaster
19/11/11 09:53:25 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/11/11 09:53:25 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/11/11 09:53:25 INFO DiskBlockManager: Created local directory at /private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/blockmgr-f239aac2-9b34-490c-a65e-ec96ef9b1087
19/11/11 09:53:25 INFO MemoryStore: MemoryStore started with capacity 434.4 MB
19/11/11 09:53:25 INFO SparkEnv: Registering OutputCommitCoordinator
19/11/11 09:53:25 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/11/11 09:53:25 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.0.12:4040
19/11/11 09:53:25 INFO SparkContext: Added JAR file:///Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.4.jar at spark://172.16.0.12:58107/jars/spark-examples_2.11-2.4.4.jar with timestamp 1573437205747
19/11/11 09:53:25 INFO SparkContext: Added JAR file:///Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7/examples/jars/scopt_2.11-3.7.0.jar at spark://172.16.0.12:58107/jars/scopt_2.11-3.7.0.jar with timestamp 1573437205747
19/11/11 09:53:25 INFO Executor: Starting executor ID driver on host localhost
19/11/11 09:53:25 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58110.
19/11/11 09:53:25 INFO NettyBlockTransferService: Server created on 172.16.0.12:58110
19/11/11 09:53:25 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/11/11 09:53:25 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.16.0.12, 58110, None)
19/11/11 09:53:25 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.0.12:58110 with 434.4 MB RAM, BlockManagerId(driver, 172.16.0.12, 58110, None)
19/11/11 09:53:25 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.16.0.12, 58110, None)
19/11/11 09:53:25 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.0.12, 58110, None)
19/11/11 09:53:26 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
19/11/11 09:53:26 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 10 output partitions
19/11/11 09:53:26 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
19/11/11 09:53:26 INFO DAGScheduler: Parents of final stage: List()
19/11/11 09:53:26 INFO DAGScheduler: Missing parents: List()
19/11/11 09:53:26 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
19/11/11 09:53:26 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1936.0 B, free 434.4 MB)
19/11/11 09:53:26 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1256.0 B, free 434.4 MB)
19/11/11 09:53:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.16.0.12:58110 (size: 1256.0 B, free: 434.4 MB)
19/11/11 09:53:26 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
19/11/11 09:53:26 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
19/11/11 09:53:26 INFO TaskSchedulerImpl: Adding task set 0.0 with 10 tasks
19/11/11 09:53:26 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, executor driver, partition 4, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, executor driver, partition 5, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, executor driver, partition 6, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, executor driver, partition 7, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, PROCESS_LOCAL, 7866 bytes)
19/11/11 09:53:26 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
19/11/11 09:53:26 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
19/11/11 09:53:26 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
19/11/11 09:53:26 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
19/11/11 09:53:26 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
19/11/11 09:53:26 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
19/11/11 09:53:26 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
19/11/11 09:53:26 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)
19/11/11 09:53:26 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
19/11/11 09:53:26 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
19/11/11 09:53:26 INFO Executor: Fetching spark://172.16.0.12:58107/jars/scopt_2.11-3.7.0.jar with timestamp 1573437205747
19/11/11 09:53:26 INFO TransportClientFactory: Successfully created connection to /172.16.0.12:58107 after 35 ms (0 ms spent in bootstraps)
19/11/11 09:53:26 INFO Utils: Fetching spark://172.16.0.12:58107/jars/scopt_2.11-3.7.0.jar to /private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/spark-ecf82630-e21a-4a28-804e-a5ee8b0d232b/userFiles-39099898-0a52-4477-9152-85c2bd301a82/fetchFileTemp11045881112579144176.tmp
19/11/11 09:53:26 INFO Executor: Adding file:/private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/spark-ecf82630-e21a-4a28-804e-a5ee8b0d232b/userFiles-39099898-0a52-4477-9152-85c2bd301a82/scopt_2.11-3.7.0.jar to class loader
19/11/11 09:53:26 INFO Executor: Fetching spark://172.16.0.12:58107/jars/spark-examples_2.11-2.4.4.jar with timestamp 1573437205747
19/11/11 09:53:26 INFO Utils: Fetching spark://172.16.0.12:58107/jars/spark-examples_2.11-2.4.4.jar to /private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/spark-ecf82630-e21a-4a28-804e-a5ee8b0d232b/userFiles-39099898-0a52-4477-9152-85c2bd301a82/fetchFileTemp15926653109345999028.tmp
19/11/11 09:53:26 INFO Executor: Adding file:/private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/spark-ecf82630-e21a-4a28-804e-a5ee8b0d232b/userFiles-39099898-0a52-4477-9152-85c2bd301a82/spark-examples_2.11-2.4.4.jar to class loader
19/11/11 09:53:27 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 867 bytes result sent to driver
19/11/11 09:53:27 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 867 bytes result sent to driver
19/11/11 09:53:27 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 867 bytes result sent to driver
19/11/11 09:53:27 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 867 bytes result sent to driver
19/11/11 09:53:27 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 867 bytes result sent to driver
19/11/11 09:53:27 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 867 bytes result sent to driver
19/11/11 09:53:27 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 711 ms on localhost (executor driver) (1/10)
19/11/11 09:53:27 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 910 bytes result sent to driver
19/11/11 09:53:27 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 867 bytes result sent to driver
19/11/11 09:53:27 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 717 ms on localhost (executor driver) (2/10)
19/11/11 09:53:27 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 867 bytes result sent to driver
19/11/11 09:53:27 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 714 ms on localhost (executor driver) (3/10)
19/11/11 09:53:27 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 713 ms on localhost (executor driver) (4/10)
19/11/11 09:53:27 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 867 bytes result sent to driver
19/11/11 09:53:27 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 714 ms on localhost (executor driver) (5/10)
19/11/11 09:53:27 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 716 ms on localhost (executor driver) (6/10)
19/11/11 09:53:27 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 717 ms on localhost (executor driver) (7/10)
19/11/11 09:53:27 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 717 ms on localhost (executor driver) (8/10)
19/11/11 09:53:27 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 718 ms on localhost (executor driver) (9/10)
19/11/11 09:53:27 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 738 ms on localhost (executor driver) (10/10)
19/11/11 09:53:27 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/11/11 09:53:27 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 0.954 s
19/11/11 09:53:27 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.027161 s
Pi is roughly 3.137619137619138
19/11/11 09:53:27 INFO SparkUI: Stopped Spark web UI at http://172.16.0.12:4040
19/11/11 09:53:27 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/11/11 09:53:27 INFO MemoryStore: MemoryStore cleared
19/11/11 09:53:27 INFO BlockManager: BlockManager stopped
19/11/11 09:53:27 INFO BlockManagerMaster: BlockManagerMaster stopped
19/11/11 09:53:27 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/11/11 09:53:27 INFO SparkContext: Successfully stopped SparkContext
19/11/11 09:53:27 INFO ShutdownHookManager: Shutdown hook called
19/11/11 09:53:27 INFO ShutdownHookManager: Deleting directory /private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/spark-ecf82630-e21a-4a28-804e-a5ee8b0d232b
19/11/11 09:53:27 INFO ShutdownHookManager: Deleting directory /private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/spark-245daa7b-9844-4d5c-958a-ee63cd378f63

IDEA配置

quick-start

基础

bash-3.2$ $SPARK_HOME/bin/spark-shell
19/11/11 10:57:53 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://172.16.0.12:4040
Spark context available as 'sc' (master = local[*], app id = local-1573441079226).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_211)
Type in expressions to have them evaluated.
Type :help for more information.

scala> val textFile = spark.read.textFile("/Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7/README.md")
textFile: org.apache.spark.sql.Dataset[String] = [value: string]

scala> textFile.count()
res0: Long = 105

scala> textFile.first()
res1: String = # Apache Spark

scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
linesWithSpark: org.apache.spark.sql.Dataset[String] = [value: string]

scala> textFile.filter(line =>line.contains("Spark")).count()
res2: Long = 20

更多操作

scala> textFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b)
res3: Int = 22

scala> import java.lang.Math
import java.lang.Math

scala> textFile.map(line => line.split(" ").size).reduce((a, b) => Math.max(a, b))
res4: Int = 22

scala> val wordCounts = textFile.flatMap(line => line.split(" ")).groupByKey(identity).count()
wordCounts: org.apache.spark.sql.Dataset[(String, Long)] = [value: string, count(1): bigint]

scala> wordCounts.collect()
res5: Array[(String, Long)] = Array((online,1), (graphs,1), (["Parallel,1), (["Building,1), (thread,1), (documentation,3), (command,,2), (abbreviated,1), (overview,1), (rich,1), (set,2), (-DskipTests,1), (name,1), (page](http://spark.apache.org/documentation.html).,1), (["Specifying,1), (stream,1), (run:,1), (not,1), (programs,2), (tests,2), (./dev/run-tests,1), (will,1), ([run,1), (particular,2), (option,1), (Alternatively,,1), (by,1), (must,1), (using,5), (you,4), (MLlib,1), (DataFrames,,1), (variable,1), (Note,1), (core,1), (more,1), (protocols,1), (guidance,2), (shell:,2), (can,7), (site,,1), (systems.,1), (Maven,1), ([building,1), (configure,1), (for,12), (README,1), (Interactive,2), (how,3), ([Configuration,1), (Hive,2), (system,1), (provides,1), (Hadoop-supported,1), (pre-built,1...

caching

scala> val textFile = spark.read.textFile("/Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7/README.md")
textFile: org.apache.spark.sql.Dataset[String] = [value: string]

scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
linesWithSpark: org.apache.spark.sql.Dataset[String] = [value: string]

scala> linesWithSpark.cache()
res0: linesWithSpark.type = [value: string]
 

scala> linesWithSpark.count()
res2: Long = 20

scala> linesWithSpark.count()
res3: Long = 20

第二次速度比第一次速度更快。

WorldCount例子

maven加入依赖:

   <!-- Spark dependency -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>2.4.4</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>2.4.4</version>
        </dependency>


        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.12.10</version>
        </dependency>

程序:

package com.xxx.spark.demo.day01

import org.apache.spark.{SparkConf, SparkContext}


/**
  * 功能：
  *
  * @author linhongz
  * @date 2019-11-11 14:42
  *
  */
object WordCount extends App {

  val conf = new SparkConf().setAppName("word count").setMaster("local")
  val sc = new SparkContext(conf)
  sc.setLogLevel("ERROR")

  val rdd = sc.textFile("/Users/linhongzheng/software/spark-2.4.4-bin-hadoop2.7/README.md") //设置操作的文本
    .flatMap(_.split(' ')) //空格分割
    .filter(x => (!"".equals(x))) //过滤掉空格
    .map((_, 1)) //(world,1),(Hello,1),(Hello,1),
    .reduceByKey(_ + _) //(world,1),(Hello,2)
    .sortBy(x => (x._2), false) //(Hello,2),(world,1)排序
    .foreach(print(_)) //输出

  sc.stop()
}

打包运行

将工程代码打包成jar包发送到远端或者本地的spark集群上运行。打包有以下步骤:

1. 点击“File“然后选择“project Structure“

2. 如图所示进行如下操作

3. 在弹出的对话框中点击按钮，选择主类进行如下操作。

由于我们的jar包实在spark上运行的，所可以删除其他不需要的依赖包，如下图所示，删除其他不需要的包，只留下红色矩形中的两个。

注意：output directory 的路径。此处是你导出 jar 的路径。

4. 执行 bulid 构建jar
   

5. 运行job

./bin/spark-submit  --class com.xxx.spark.demo.day01.WordCount --master local /Users/linhongzheng/workstation/spark/sparkDemo/out/artifacts/sparkDemo_jar/sparkDemo.jar

报下面的异常:

9/11/11 17:21:09 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.0.12, 53994, None)
Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction2$mcIII$sp
	at com.xxx.spark.demo.day01.WordCount$.main(WordCount.scala:26)
	at com.xxx.spark.demo.day01.WordCount.main(WordCount.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction2$mcIII$sp
	... 14 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.java8.JFunction2$mcIII$sp
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more

spark安装目录下的jars中依赖scala是2.11版本的：

但官网写着是2.12版本，有点坑：

在下载Spark页面提示只有2.4.2是基于Scala2.12编译

所以需要重新下载2.4.2安装包，地址：https://archive.apache.org/dist/spark/spark-2.4.2/

19/11/11 17:41:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/11/11 17:41:35 INFO SparkContext: Running Spark version 2.4.2
19/11/11 17:41:35 INFO SparkContext: Submitted application: word count
19/11/11 17:41:35 INFO SecurityManager: Changing view acls to: linhongzheng
19/11/11 17:41:35 INFO SecurityManager: Changing modify acls to: linhongzheng
19/11/11 17:41:35 INFO SecurityManager: Changing view acls groups to:
19/11/11 17:41:35 INFO SecurityManager: Changing modify acls groups to:
19/11/11 17:41:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(linhongzheng); groups with view permissions: Set(); users  with modify permissions: Set(linhongzheng); groups with modify permissions: Set()
19/11/11 17:41:35 INFO Utils: Successfully started service 'sparkDriver' on port 54482.
19/11/11 17:41:35 INFO SparkEnv: Registering MapOutputTracker
19/11/11 17:41:35 INFO SparkEnv: Registering BlockManagerMaster
19/11/11 17:41:35 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/11/11 17:41:35 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/11/11 17:41:35 INFO DiskBlockManager: Created local directory at /private/var/folders/rq/kl4ybn097tzbfbwr91cwv_640000gn/T/blockmgr-0aaa0d8b-9ce8-416c-81c5-fcb8ae50e3cb
19/11/11 17:41:35 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/11/11 17:41:35 INFO SparkEnv: Registering OutputCommitCoordinator
19/11/11 17:41:35 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/11/11 17:41:35 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.0.12:4040
19/11/11 17:41:35 INFO SparkContext: Added JAR file:/Users/linhongzheng/workstation/spark/sparkDemo/out/artifacts/sparkDemo_jar/sparkDemo.jar at spark://172.16.0.12:54482/jars/sparkDemo.jar with timestamp 1573465295557
19/11/11 17:41:35 INFO Executor: Starting executor ID driver on host localhost
19/11/11 17:41:35 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54483.
19/11/11 17:41:35 INFO NettyBlockTransferService: Server created on 172.16.0.12:54483
19/11/11 17:41:35 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/11/11 17:41:35 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.16.0.12, 54483, None)
19/11/11 17:41:35 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.0.12:54483 with 366.3 MB RAM, BlockManagerId(driver, 172.16.0.12, 54483, None)
19/11/11 17:41:35 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.16.0.12, 54483, None)
19/11/11 17:41:35 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.0.12, 54483, None)
(the,24)(to,17)(Spark,16)(for,12)(and,10)(##,9)(a,9)(is,7)(run,7)(on,7)(can,7)(in,6)(using,5)(of,5)(also,5)(if,4)(build,4)(including,4)(Please,4)(see,4)(an,4)(You,4)(with,4)(you,4)(For,3)(use,3)(or,3)(Hadoop,3)(general,3)(documentation,3)(example,3)(how,3)(one,3)(This,2)(Hive,2)(To,2)(SparkPi,2)(refer,2)(Interactive,2)(be,2)(./bin/run-example,2)(1000:,2)(tests,2)(examples,2)(at,2)(Shell,2)(class,2)(`examples`,2)(set,2)(Hadoop,,2)(cluster,2)(supports,2)(particular,2)(Python,2)(locally,2)(following,2)(which,2)(should,2)(do,2)(It,2)(Scala,2)(detailed,2)(return,2)(Python,,2)(building,2)(guidance,2)(SQL,2)(that,2)(programs,2)(shell:,2)(command,,2)(package,1)(Programs,1)(processing.,1)(Because,1)(The,1)(page](http://spark.apache.org/documentation.html).,1)(cluster.,1)(its,1)([run,1)(than,1)(APIs,1)(have,1)(Try,1)(computation,1)(through,1)(several,1)(graph,1)(storage,1)(["Specifying,1)("yarn",1)(Once,1)(["Useful,1)(prefer,1)(engine,1)(version,1)(file,1)(documentation,,1)(processing,,1)(are,1)(systems.,1)(params,1)(not,1)(different,1)(R,,1)(given.,1)(when,1)(Tests,1)(Apache,1)(thread,1)(programs,,1)(Spark.,1)(package.,1)(1000).count(),1)(Versions,1)(HDFS,1)(Data.,1)(>>>,1)(Maven,1)(programming,1)(Testing,1)(module,,1)(Streaming,1)(environment,1)(run:,1)(test,,1)(clean,1)(rich,1)(Developer,1)(Version,1)(GraphX,1)(guide](http://spark.apache.org/contributing.html),1)(URL,,1)(threads.,1)(same,1)(MASTER=spark://host:7077,1)(built,1)(against,1)([Apache,1)(optimized,1)(3"](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3).,1)(usage,1)(development,1)(Maven,,1)(graphs,1)(talk,1)(Enabling,1)(abbreviated,1)(directory.,1)(README,1)(computing,1)(overview,1)(example:,1)(N,1)(Hadoop-supported,1)(running,1)(find,1)(contains,1)(project,1)(Pi,1)(need,1)(Big,1)(high-level,1)(Java,,1)(uses,1)(<class>,1)(available,1)(requires,1)((You,1)(more,1)(Documentation,1)(tools,1)(using:,1)(must,1)(built,,1)(tests](http://spark.apache.org/developer-tools.html#individual-tests).,1)(system,1)(build/mvn,1)(this,1)(integration,1)(There,1)(YARN,,1)(pre-built,1)([Configuration,1)(library,1)(A,1)(locally.,1)(sc.parallelize(1,1)(only,1)(Configuration,1)(basic,1)(#,1)(changed,1)(More,1)(learning,,1)(first,1)(./bin/pyspark,1)(info,1)([params]`.,1)([project,1)(mesos://,1)(Maven](http://maven.apache.org/).,1)(setup,1)(<http://spark.apache.org/>,1)(latest,1)(your,1)(MASTER,1)(["Parallel,1)(scala>,1)(DataFrames,,1)(provides,1)(configure,1)(distributions.,1)(About,1)(instructions.,1)(easiest,1)(no,1)(project.,1)(`./bin/run-example,1)(started,1)(Note,1)(by,1)(individual,1)(spark://,1)(tips,,1)(Alternatively,,1)(variable,1)(submit,1)(-T,1)(machine,1)(thread,,1)(them,,1)(stream,1)(And,1)(distribution,1)(review,1)(Thriftserver,1)(developing,1)(./bin/spark-shell,1)("local",1)(start,1)(Spark](#building-spark).,1)(help,1)(print,1)(Spark"](http://spark.apache.org/docs/latest/building-spark.html).,1)(Kubernetes,1)(Contributing,1)(-DskipTests,1)(data,1)(versions,1)(online,1)(downloaded,1)(Guide](http://spark.apache.org/docs/latest/configuration.html),1)(builds,1)(comes,1)(Tools"](http://spark.apache.org/developer-tools.html).,1)([building,1)(Many,1)(Running,1)(from,1)(way,1)(Online,1)(site,,1)(other,1)(Example,1)([Contribution,1)(analysis.,1)(sc.parallelize(range(1000)).count(),1)(runs.,1)(Building,1)(higher-level,1)(protocols,1)(guide,,1)(name,1)(fast,1)(will,1)(IDE,,1)(get,1)(core,1)(information,1)(instance:,1)(YARN"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn),1)(web,1)(resource-managers/kubernetes/integration-tests/README.md,1)("local[N]",1)(option,1)(MLlib,1)(["Building,1)(contributing,1)(Scala,,1)(package.),1)(./dev/run-tests,1)(sample,1)bash-3.2$