Spark编程

val sc = new SparkContext(arg(0),"WordCount", System.getenv("SPARK_HOME"),Seq(System.getenv("SPARK_TEST_JAR")))
val textFile=sc.textFile(arg(1))
val input FormatClass = classOf(SequenceFileInputFormat[text,text]]
var hadoopRdd=sc.hadoopRDD(conf,inputFormatClass, classOf[text],classOf[text])

val result = hadoopRdd.flatMap{case(key,value) => value.toString().split("\\s+");}.map(word => (word,1)).reduceByKey(_+_)

Spark 机器学习的数据集：

http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/