linux_ja的专栏

执行： ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://192.168.0.63:7077 --executor-memory 20G --total-executor-cores 100 examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.

val file = sc.textFile("hdfs://datanode3:8020/test/word.txt") val count = file.flatMap(line => line.split(",")).map(word => (word, 1)).reduceByKey(_+_) count.collect()

Loss was due to java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dposv

/** * 1.0 1.0 * 1.0 2.0 * 2.0 1.0 * 2.0 2.0 * 3.0 3.0 * 3.0 4.0 * 4.0 3.0 * 4.0 4.0 */package com.spark.testimport org.apache.spark.{ SparkContext, SparkConf }import org.apache.spark.Spa

package com.spark.testimport org.apache.spark.{ SparkConf, SparkContext }import org.apache.spark.sql.SQLContextcase class Person(name: String, age: Int) extends java.io.Serializable object Peop

用sbt构建json-lib失败，以下是配置方法：libraryDependencies += "net.sf.json-lib" % "json-lib" % "2.4" from "http://repo1.maven.org/maven2/net/sf/json-lib/json-lib/2.4/json-lib-2.4-jdk15.jar"

jmx监控spark比storm稍微有点繁琐：首先在spark-defaults.conf中添加 ，但是8711端口不能重复，也就是说不能在一个节点上启动两个executor，或者端口冲突，没有storm友好 spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dc

1、报：Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient是由于元数据默认到derby中找，所以提供mysql的解决：在spar