jim8973的博客

1、Tuple类型val products = sc.parallelize(List("屏保 20 10","支架 20 1000","酒精棉 5 2000","吸氧机 5000 1000")) val productData = products.map(x=>{ val splits = x.split(" ") val name = splits(0) val price = splits(1).toDouble val amount

例子object Work02App { def main(args: Array[String]): Unit = { val sparkConf = new SparkConf().setMaster("local[2]").setAppName(this.getClass.getSimpleName) val sc = new SparkContext(sparkCon...

1、在本地创建zip文件# 在spark的jars目录下zip spark.zip ./*2、HDFS上创建存放spark jar目录hdfs dfs -mkdir -p /spark-yarn/jars3、将$SPARK_HOME/jars下的spark.zip包上传至刚健的HDFS路径[hadoop@hadoop jars]$ hdfs dfs -put ./spark.zip...

概述spark on yarn 的优势是每个Spark executor作为一个Yarn容器(container)运行，Spark可以使得多个Task在同一个容器(container)里面运行；有两种模式：yarn-client和yarn-clusteryarn-client:driver运行在client端；适用于调式，能直接看到各种日志；连接断了，任务就挂了driver运行在am中；日......

官方部署文档：http://spark.apache.org/docs/latest/building-spark.htmlspark的github源码下载地址：https://github.com/apache/spark设置maven的执行内存：export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g"打包：./build/mvn -...

数据www.baidu.com,url5www.baidu.com,url5www.baidu.com,url5www.baidu.com,url5www.baidu.com,url5www.google.com,url8www.google.com,url2www.google.com,url1www.twitter.com,url10www.twitter.com,url1...

zip:拉链，分区数需要相同（分区不同Can’t zip RDDs with unequal numbers of partitions: List(4, 2)）；元素个数也要相同（Can only zip RDDs with same number of elements in each partition） val rddzip1 = sc.parallelize(List("ruoze......

**org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 2) had a not serializable result:object not serializable (class: org.apache.hadoop.hbase.io.ImmutableB...