Spark Dataframe Scala选择部分列

最新推荐文章于 2025-12-10 13:14:14 发布

原创最新推荐文章于 2025-12-10 13:14:14 发布 · 118 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #scala #大数据 #分布式 #开发语言

本文介绍了如何使用Scala DataFrame进行列筛选，通过两种方法实现：一是利用DataFrame的列名过滤，二是运用正则表达式匹配。展示了如何根据不包含特定字符的列名选择数据。

一,创建Dataframe
scala> val df = sc.parallelize(Seq(
     |      |   (0,"cat26",30.9),
     |      |   (1,"cat67",28.5),
     |      |   (2,"cat56",39.6),
     |      |   (3,"cat8",35.6))).toDF("Hour", "Category", "Value")
df: org.apache.spark.sql.DataFrame = [Hour: int, Category: string ... 1 more field]

scala> df.show()
+----+--------+-----+
|Hour|Category|Value|
+----+--------+-----+
|   0|   cat26| 30.9|
|   1|   cat67| 28.5|
|   2|   cat56| 39.6|
|   3|    cat8| 35.6|
+----+--------+-----+

二,方法1:(!号是取反)
scala> var df1 =  df.select(df.columns.filter(x => !x.contains("Val")).map(df(_)) : _*)
df1: org.apache.spark.sql.DataFrame = [Hour: int, Category: string]
scala> df1.show()
+----+--------+
|Hour|Category|
+----+--------+
|   0|   cat26|
|   1|   cat67|
|   2|   cat56|
|   3|    cat8|
+----+--------+

三,方法2:
scala> val regex = """^((?!Va).)*$""".r
regex: scala.util.matching.Regex = ^((?!Va).)*$

scala> val selection = df.columns.filter(s => regex.findFirstIn(s).isDefined)
selection: Array[String] = Array(Hour, Category)

scala> var newdf = df.select(selection.head, selection.tail : _*)
newdf: org.apache.spark.sql.DataFrame = [Hour: int, Category: string]

scala> newdf.show()
+----+--------+
|Hour|Category|
+----+--------+
|   0|   cat26|
|   1|   cat67|
|   2|   cat56|
|   3|    cat8|
+----+--------+

正则表达式这块没怎么研究,可参考:
https://www.runoob.com/scala/scala-regular-expressions.html
https://stackoverflow.com/questions/59065137/select-columns-in-spark-dataframe-based-on-column-name-pattern