该文主要展示的是spark sql 例子
(内容是找了份oracle的例子,翻译成spark sql的)
1、需要准备好四张表,既四个文本文件逗号分隔
2、为这四张表创建好schema,并注册成表
3、时间处理有小部分改动
准备的四张表
image.png
image.png
image.png
image.png
四张表中的数据
Teacher
884,zhagnxueyou,female,1980/2/10,Assistant professor,department of computer
856,liudehua,female,1980/12/10,professor,department of electronic engineering
825,liming,female,1980/1/10,advandced professor,department of computer
831,zhourunfa,female,1981/12/10,jonior professor,department of electronic engineering
Student
108,ceng hua,male,1977/9/1,95033
105,zhangsan,male,1975/10/22,95031
107,wangli,female,1976/1/8,95033
101,lili,male,1978/5/14,95033
109,wangfang,female,1979/5/10,95031
103,lijun,male,1974/5/21,95031
Score
103,3-245,86
105,3-245,75
109,3-245,68
103,3-105,92
105,3-105,88
109,3-105,76
101,3-105,64
107,3-105,91
108,3-105,78
101,6-166,85
107,6-166,79
108,6-166,81
Course
3-105,introduction to computer,825
3-245,the operating system,804
6-166,the digital circuit,856
9-888,higher math,831
maven
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
</dependency>
例子代码,粘贴可用,注意注释掉不需要的地方即可
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.types._
import scala.collection.mutable
import java.text.SimpleDateFormat
/**
* 2017/05/26
* CDP
*/
object SparkSqlExample1 {
def main(args: Array[String]): Unit = {
/** ***************************************************************************************************************
* sparksession
*/
val spark = SparkSession
.builder()
.master("local")
.appName("test")
.config("spark.sql.shuffle.partitions", "5")
.getOrCreate()
/** ***************************************************************************************************************
* 表结构
*/
val StudentSchema: StructType = StructType(mutable.ArraySeq( //学生表
StructField("Sno", StringType, nullable = false), /