WIN10配置Spark+IDEA+Maven开发环境

1、配置IDEA+Maven+Scala开发环境

https://yellow520.blog.youkuaiyun.com/article/details/112616105

2、配置WIN10上Hadoop开发环境

https://yellow520.blog.youkuaiyun.com/article/details/115712233

配好后，%HADOOP_HOME%\bin就有winutils.exe了，
可解决ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

3、pom.xml 添加 Spark

1. 去【https://mvnrepository.com/】找对应版本
   

2. 复制到pom.xml

<properties>
    <spark.version>3.0.2</spark.version>
</properties>
<dependencies>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>
</dependencies>

4、关联Spark源码

1. 下载网址：https://downloads.apache.org/spark/
2. 下载源码，本文是spark-3.0.2.tgz
3. 解压到%SCALA_HOME%\lib

5、日志等级修改

1. 去到%SCALA_HOME%\lib\spark-3.0.2\conf
2. 复制log4j.properties.template到Maven工程下resources文件夹
3. 改名为log4j.properties
4. 修改日志等级log4j.rootCategory=WARN, console
5. 添加log4j-core依赖

<properties>
    <log4j-core.version>2.17.2</log4j-core.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>${log4j-core.version}</version>
    </dependency>
</dependencies>

6、试运行

object Hello {
  def main(args: Array[String]): Unit = {
    import org.apache.spark.{SparkConf, SparkContext}
    import org.apache.spark.rdd.RDD
    // 创建配置
    val conf: SparkConf = new SparkConf().setAppName("a1").setMaster("local[*]")
    // 创建文本处理对象
    val sc = new SparkContext(conf)
    // 读数据（并行化）
    val rdd0: RDD[String] = sc.parallelize(List("cat dog", "cat cow", "cow pig"))
    // 分词并平坦化
    val rdd1: RDD[String] = rdd0.flatMap(_.split(" "))
    // 每个词
    val rdd2: RDD[(String, Int)] = rdd1.map((_, 1))
    // 对相同的key进行reduce操作
    val rdd3: RDD[(String, Int)] = rdd2.reduceByKey(_ + _)
    // 打印
    println(rdd0.collect.toList)
    println(rdd1.collect.toList)
    println(rdd2.collect.toList)
    println(rdd3.collect.toList)
  }
}

打印结果

List(cat dog, cat cow, cow pig)
List(cat, dog, cat, cow, cow, pig)
List((cat,1), (dog,1), (cat,1), (cow,1), (cow,1), (pig,1))
List((cow,2), (dog,1), (pig,1), (cat,2))

7、打包+提交到集群

1. pom.xml加入下面Scala打包插件

<build>
    <finalName>Hello</finalName>
    <plugins>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>3.4.6</version>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

2. 右边栏点击打包
   

3. 将Hello.jar上传到Linux服务器
   

4. Linux上先装好Hadoop和Spark才能提交，命令如下