WIN10配置Spark+IDEA+Maven开发环境

原创

已于 2022-04-15 19:08:01 修改 · 711 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#log4j #spark #maven

于 2021-04-29 14:13:22 首次发布

发出火花的目录

1、配置IDEA+Maven+Scala开发环境
2、配置WIN10上Hadoop开发环境
3、pom.xml 添加 Spark
4、关联Spark源码
5、日志等级修改
6、试运行
7、打包+提交到集群

1、配置IDEA+Maven+Scala开发环境

https://yellow520.blog.youkuaiyun.com/article/details/112616105

2、配置WIN10上Hadoop开发环境

https://yellow520.blog.youkuaiyun.com/article/details/115712233

配好后，%HADOOP_HOME%\bin就有winutils.exe了，
可解决ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

3、pom.xml 添加 Spark

去【https://mvnrepository.com/】找对应版本
复制到pom.xml

<properties>
    <spark.version>3.0.2</spark.version>
</properties>
<dependencies>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>${spark.version}</version>
    </dependency>