stupidTomA的博客

官网上介绍的自定义聚合例子http://spark.apache.org/docs/latest/sql-getting-started.html#aggregationsSparkSQL的聚合函数支持count(), countDistinct(), avg(), max(), min()等，但是肯定不满足，所以需要用户自定义函数第一种 弱类型定义 extend UserDefi...

import org.apache.spark.SparkConfimport org.apache.spark.sql.SparkSessionobject SparkSqlDemo { def main(args: Array[String]): Unit = { val sparkConf = new SparkConf().setAppName("demo").setM...

import org.apache.spark.SparkConfimport org.apache.spark.sql.SparkSessionobject Transform { case class People(name:String,age:Int) def main(args: Array[String]): Unit = { val sparkConf =...

1.配置jdk 1.82.配置scala scala-2.12.113.下载spark， 要和上面的Scala相对应，不然idea里面的醒目启动的时候会报错4.下载hadoop 2.7.7 https://archive.apache.org/dist/hadoop/common/，配置HADOOP_HOME，并下载hadoop版本对应的winutils.exe 覆...

spark-submit --class wordcount --master spark://aml168:7077 --driver-memory 800M --executor-memory 600M -...

1.搭建好StandarOne之后，写了一个例子，将jar包上传到节点上去，运行任务19/11/26 14:52:50 ERROR TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting jobException in thread "main" org.apache.spark.SparkException: Job abo...