dataframe scala 修改值_Scala 对DataFrame 列名重命名处理

本文介绍常见的几种对spark dataframe 进行列名的修改方法

生成测试数据

val data = Seq(Row(Row("James ","","Smith"),"36636","M",3000),

Row(Row("Michael ","Rose",""),"40288","M",4000),

Row(Row("Robert ","","Williams"),"42114","M",4000),

Row(Row("Maria ","Anne","Jones"),"39192","F",4000),

Row(Row("Jen","Mary","Brown"),"","F",-1)

)

定义schema

val schema = new StructType()

.add("name",new StructType()

.add("firstname",StringType)

.add("middlename",StringType)

.add("lastname",StringType))

.add("dob",StringType)

.add("gender",StringType)

.add("salary",IntegerType)

创建 DataFrame

val df = spark.createDataFrame(spark.sparkContext.parallelize(data),schema)

df.printSchema()

下图是打印schema信息

1.使用 withColumnRenamed 重命名

**`withColumnRenamed()` 函数是最直接的方法，第一个参数是原来的名字，第二个是新名字**

**Syntax:**

def withColumnRenamed(existingName: String, newName: String): DataFrame

**`existingName` – 存在的原始的名字**

**`newName` – 新的列名**

返回一个新的 DataFrame**.**

举例**:**

df.withColumnRenamed("dob","DateOfBirth")

.printSchema()

上面的例子就是将 dob

2. 使用 withColumnRenamed 重命名多列

val df2 = df.withColumnRenamed("dob","DateOfBirth")

.withColumnRenamed("salary","salary_amount")

df2.printSchema()

3. 使用 StructType 结构体实现嵌套命名

val schema2 = new StructType()

.add("fname",StringType)

.add("middlename",StringType)

.add("lname",StringType)

df.select(col("name").cast(schema2),

col("dob"),

col("gender"),

col("salary"))

.printSchema()

上面的例子就是讲name变成的了嵌套的名字包含 firstname middlename lastname

4. 使用 Select 修改嵌套命名

df.select(col("name.firstname").as("fname"),

col("name.middlename").as("mname"),

col("name.lastname").as("lname"),

col("dob"),col("gender"),col("salary"))

.printSchema()

5. 使用 withColumn 修改嵌套命名

val df4 = df.withColumn("fname",col("name.firstname"))

.withColumn("mname",col("name.middlename"))

.withColumn("lname",col("name.lastname"))

.drop("name")

df4.printSchema()

6. 使用col() 函数 – 修改全部列名或者多个列名

val old_columns = Seq("dob","gender","salary","fname","mname","lname")

val new_columns = Seq("DateOfBirth","Sex","salary","firstName","middleName","lastName")

val columnsList = old_columns.zip(new_columns).map(f=>{col(f._1).as(f._2)})

val df5 = df4.select(columnsList:_*)

df5.printSchema()

7. 使用 toDF() 修改全部的列名

当我们的列名中不包含嵌套列名的时候 **,** 给定新的 schema使用 **`toDF()`** 可以修改全部的列名

val newColumns = Seq("newCol1","newCol2","newCol3")

val df = df.toDF(newColumns:_*)