u011569805的专栏

1.wget https://github.com/apache/incubator-spark/archive/v0.8.1-incubating.zip2.mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0  -Pnew-yarn -DskipTests package3.

1. 前面都是在4台hadoop2.2集群，yi

1. scala_home= /path/2.10.32. JAVA_HOME=/path/to/

1. 直接下载编译好的 export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoopSPARK_JAR=./assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar \./bin/spark-class org.apache.spark.depl

1.读入参数构建sparkContext if (args.length < 1) { System.err.println("Usage: SparkPageRank ") System.exit(1) } val sparkConf = new SparkConf().setAppName("PageRank") val iters = if

1. 计算传递闭包（可到达路径数目）

1. spark 在做group可以做不对称de

1. 能够val sparkConf = new SparkConf().setAppName("Multi-Broadcast Test") val sc = new SparkContext(sparkConf) val slices = if (args.length > 0) args(0).toInt else 2 val num = if (args.le

1. 正则表达式，匹配下面stringval apacheLogRegex = """^([\d.]+) (\S+) (\S+) \[([\w\d:/]+\s[+\-]\d{4})\] "(.+?)" (\d{3}) ([\d\-]+) "([^"]+)" "([^"]+)".*""".r

1. 读取每一行，按 def parseVector(line: String): Vector[Double] = { DenseVector(line.split(' ').map(_.toDouble)) }

1. 生成Arraydef generateData = { def generatePoint(i: Int) = { val y = if(i % 2 == 0) -1 else 1 val x = DenseVector.fill(D){rand.nextGaussian + y * R} DataPoint(x, y) } A

1. 随机生成N个 def generateData = { def generatePoint(i: Int) = { DenseVector.fill(D){rand.nextDouble * R} } Array.tabulate(N)(generatePoint) }

1. 运行spark自带例子时遇到军

1.安装sbt并加入环境变量