SparkSQL实现分区覆盖写

最新推荐文章于 2024-07-18 22:59:23 发布

whzatsh

最新推荐文章于 2024-07-18 22:59:23 发布

阅读量4.7k

点赞数 4

分类专栏：大数据文章标签：大数据相关技术

本文链接：https://blog.youkuaiyun.com/u010585659/article/details/102898940

版权

一、创建分区表

scala> spark.sql("CREATE TABLE ptable(c1 int,c2 int) PARTITIONED BY (step string)")
res1: org.apache.spark.sql.DataFrame = []

二、向分区中插入测试数据

scala> spark.sql("INSERT INTO TABLE ptable PARTITION(step='a') select 1,2")
res2: org.apache.spark.sql.DataFrame = []

scala> spark.sql("select * from ptable")
res3: org.apache.spark.sql.DataFrame = [c1: int, c2: int ... 1 more field]

scala> spark.sql("select * from ptable").show(100,false)
+---+---+----+
|c1 |c2 |step|
+---+---+----+
|1  |2  |a   |
+---+---+----+


scala> spark.sql("INSERT INTO TABLE ptable PARTITION(step='a') select 3,4")
res5: org.apache.spark.sql.DataFrame = []

scala> spark.sql("select * from ptable").show(100,false)
+---+---+----+
|c1 |c2 |step|
+---+---+----+
|1  |2  |a   |
|3  |4  |a   |
+---+---+----+


scala> spark.sql("INSERT INTO TABLE ptable PARTITION(step='b') select 5,6")
res7: org.apache.spark.sql.DataFrame = []

scala> spark.sql("INSERT INTO TABLE ptable PARTITION(step='b') select 7,8")
res8: org.apache.spark.sql.DataFrame = []

scala> spark.sql("select * from ptable").show(100,false)
+---+---+----+
|c1 |c2 |step|
+---+---+----+
|1  |2  |a   |
|3  |4  |a   |
|5  |6  |b   |
|7  |8  |b   |
+---+---+----+


scala> spark.sql("INSERT INTO TABLE ptable PARTITION(step='c') select 9,10")
res10: org.apache.spark.sql.DataFrame = []

scala> spark.sql("INSERT INTO TABLE ptable PARTITION(step=