Spark核心编程：wordcount程序原理深度剖析

一、wordcount编码

val conf = new SparkConf().setAppName("WordCount")

val sc = new JavaSparkContext(conf)

val lines = sc.textFile("hdfs://spark1:9000/spark.txt")

val words = lines.flatMap(line => line.split(" "))

val pairs = words.map(word => (word, 1))

val wordCounts = pairs.reduceByKey(_ + _)

wordCounts.foreach(wordCount => println(wordCount._1 + " appears " + wordCount._2 + " times."))

 

二、图解原理