spark ds<->df

dataastron

于 2018-05-03 16:07:12 发布

阅读量400

点赞数

CC 4.0 BY-SA版权

分类专栏： Spark

本文链接：https://blog.youkuaiyun.com/dataastron/article/details/80181630

Spark 专栏收录该内容

14 篇文章

订阅专栏

本文介绍如何在Apache Spark中实现DataFrame(DF)与Dataset(DS)之间的转换，并通过具体示例展示了转换过程及结果。文章首先创建了一个SparkSession实例，接着读取JSON文件生成DataFrame，再将DataFrame转换为Dataset进行操作，最后又将Dataset转换回DataFrame。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import org.apache.spark.sql.SparkSession

object DF2DS1 {

  case class Employee(name: String, age: Long, depId: Long, gender: String, salary: Long)
  def main(args: Array[String]): Unit = {
    println("astron")
    val spark = SparkSession
      .builder()
      .master("local")
      .appName("star")
      .getOrCreate()
    import spark.implicits._
    // DF->DS
    val demoDf = spark.read.json("d://employee.json")
    println(demoDf)
    //demoDf.printSchema()
    val demoDs = demoDf.as[Employee]
    println(demoDs)
    //demoDs.printSchema()

    //DS->DF
    val newDemoDf = demoDs.toDF();
    println(newDemoDf)
    //newDemoDf.printSchema()
  }

astron
[age: bigint, depId: bigint ... 3 more fields]
[age: bigint, depId: bigint ... 3 more fields]
[age: bigint, depId: bigint ... 3 more fields]
root
 |-- age: long (nullable = true)
 |-- depId: long (nullable = true)
 |-- gender: string (nullable = true)
 |-- name: string (nullable = true)
 |-- salary: long (nullable = true)