Spark本地开发环境scala版本_怎么把spark的scala2.12换成2.11-优快云博客

本文链接：https://blog.youkuaiyun.com/baifanwudi/article/details/78413969

本文介绍如何使用SBT和Maven配置Spark项目，包括设置Hadoop环境变量、创建项目、添加依赖等步骤，并提供了一个简单的测试代码示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

SBT-Scala

参考上篇http://blog.youkuaiyun.com/baifanwudi/article/details/78354339
配置hadoop环境变量

新建SBT项目

这里写图片描述

sbt配置

name := "SparkScalaTest"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"

libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.0"

libraryDependencies += "org.apache.spark" % "spark-sql-kafka-0-10_2.11" % "2.2.0"

libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.7.1"

测试代码

import org.apache.spark.sql.SparkSession

object Test {
  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder().appName("LocalTest").master("local[2]").getOrCreate();
    val textFile = spark.read.textFile("README.md");
    println(textFile.count());

    println(textFile.first());

    val linesWithSpark = textFile.filter(line => line.contains("Spark"))
    val s = textFile.filter(line => line.contains("Spark")).count()
    println(s)
    spark.stop()
  }
}