IDEA创建spark maven项目并连接远程spark集群

maven-scala-plugin

2.15.0

<execution>

  <goals>

    <goal>compile</goal>

    <goal>testCompile</goal>

  </goals>

  <configuration>

    <args>

      <arg>-make:transitive</arg>

      <arg>-dependencyfile</arg>

      <arg>${project.build.directory}/.scala\_dependencies</arg>

    </args>

  </configuration>

</execution>

要去除-make:transitive这个参数，否则会报错。



3、创建SparkPi Object类

object SparkPi {

def main(args: Array[String]): Unit = {

val spark = SparkSession

  .builder

  .appName("Spark Pi")

  .master("spark://172.21.212.114:7077")

  .config("spark.jars","E:\\work\\polaris\\polaris-spark\\spark-scala\\target\\spark-scala-1.0.0.jar")

  .config("spark.e

必看视频！获取2024年最新Java开发全套学习资料 备注Java

xecutor.memory",“2g”)

  .config("spark.cores.max","2")

  .config("spark.driver.host", "172.21.58.28")

  .config("spark.driver.port", "9089")

  .getOrCreate()

//spark = new SparkContext(conf).

val slices = if (args.length > 0) args(0).toInt else 2

val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow

val count = spark.sparkContext.parallelize(1 until n, slices).map { i =>

  val x = random * 2 - 1

  val y = random * 2 - 1

  if (x*x + y*y <= 1) 1 else 0

}.reduce(_ + _)

println(s"Pi is roughly ${4.0 * count / (n - 1)}")

spark.stop()

}

}

4、执行打包命令：



![](https://img-blog.csdnimg.cn/20210830150945956.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAUG9sYXJpc0h1c3Rlcg==,size_10,color_FFFFFF,t_70,g_se,x_16)



5、点击Idea Run执行即可：



![](https://img-blog.csdnimg.cn/2021083015112631.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAUG9sYXJpc0h1c3Rlcg==,size_9,color_FFFFFF,t_70,g_se,x_16)






# 独家面经总结，超级精彩

本人面试腾讯，阿里，百度等企业总结下来的面试经历，都是真实的，分享给大家！

![image](https://img-blog.csdnimg.cn/img_convert/061490d32aa9c272bdbc47101b9ecdf2.webp?x-oss-process=image/format,png)

![image](https://img-blog.csdnimg.cn/img_convert/2518241fd555f751711a1c611ff9d076.webp?x-oss-process=image/format,png)

![image](https://img-blog.csdnimg.cn/img_convert/5b99aac782d618c5f9643e4658775699.webp?x-oss-process=image/format,png)

![image](https://img-blog.csdnimg.cn/img_convert/319570a67909e026ba081a3eff8458f9.webp?x-oss-process=image/format,png)

# Java面试准备

准确的说这里又分为两部分：

1.  Java刷题
2.  算法刷题

Java刷题：此份文档详细记录了千道面试题与详解；

![image](https://img-blog.csdnimg.cn/img_convert/7718017f6ca62e613139409a2d7adf71.webp?x-oss-process=image/format,png)

![image](https://img-blog.csdnimg.cn/img_convert/14d7367cc88b013156bfdffc94128a19.webp?x-oss-process=image/format,png)

里又分为两部分：

1.  Java刷题
2.  算法刷题

Java刷题：此份文档详细记录了千道面试题与详解；

[外链图片转存中...(img-VogQ79qn-1716444230985)]

[外链图片转存中...(img-f4Howbn4-1716444230985)]