scala , spark ,sbt-优快云博客

本文介绍了一个使用Scala编写的Spark应用程序示例。该程序通过读取文件并计算包含字符'a'和'b'的行数，展示了如何配置Spark环境、创建Spark上下文及执行基本的数据过滤与计数操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

2019独角兽企业重金招聘Python工程师标准>>>

cat simple.sbt

name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.2"
libraryDependencies += "org.scala-debugger" %% "scala-debugger-api" % "1.0.0"

cat src/main/scala/simple.scala

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp{
    def main(args: Array[String]) {
        val logFile = "file:///dev/shm/hello.scala"
        val conf = new SparkConf().setAppName("Lambda")
        val sc = new SparkContext(conf)
        val logData = sc.textFile(logFile,2).cache()

        val numAs = logData.filter(line => line.contains("a")).count()
        val numBs = logData.filter(line => line.contains("b")).count()

        println("Lines with a %s  Line with b %s".format(numAs, numBs))
        sc.stop()
    }
}

/opt/java/spark-2.0.1-bin-hadoop2.7/bin/spark-submit --class SimpleApp target/scala-2.11/simple-project_2.11-1.0.jar

转载于:https://my.oschina.net/innovation/blog/806959