spark09-wordcount程序执行原理

val conf: SparkConf = new SparkConf().setMaster("local").setAppName("wordcount")
val context: SparkContext = new SparkContext(conf)
val file: RDD[String] = context.textFile("datas")
val wordcount: RDD[(String, Int)] = file.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
wordcount.collect().foreach(println(_))
context.stop()

1.应用程序会new SparkContext对象

SparkConf 环境配置

SparkEnv 环境

SchedulerBackend 调度后台

TaskScheduler 任务调度

DAGScheduler 作业调度

2.构建依赖

context.textFile("datas")

flatMap(_.split(" "))