193、Spark 2.0之Dataset开发详解-typed操作：joinWith

最新推荐文章于 2023-07-29 17:48:09 发布

ZFH__ZJ

最新推荐文章于 2023-07-29 17:48:09 发布

阅读量1.2k

点赞数

CC 4.0 BY-SA版权

分类专栏： Spark入坑

本文链接：https://blog.youkuaiyun.com/ZJ__ZFH/article/details/88059527

Spark入坑专栏收录该内容

207 篇文章

订阅专栏

本文通过一个具体的代码示例，展示了如何使用Apache Spark的Dataset API中的joinWith方法进行数据连接操作，对比join的区别在于连接后新Dataset的schema变化。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

和join的区别是连接后的新Dataset的schema会不一样

代码

object TypedOperation {

  case class Employee(name: String, age: Long, depId: Long, gender: String, salary: Long)

  case class Department(id: Long, name: String)

  def main(args: Array[String]): Unit = {
    val sparkSession = SparkSession
      .builder()
      .appName("BasicOperation")
      .master("local")
      .getOrCreate()

    import sparkSession.implicits._
    import org.apache.spark.sql.functions._

    val employeePath = this.getClass.getClassLoader.getResource("employee.json").getPath
    val departmentPath = this.getClass.getClassLoader.getResource("department.json").getPath

    val employeeDF = sparkSession.read.json(employeePath)
    val departmentDF = sparkSession.read.json(departmentPath)

    val employeeDS = employeeDF.as[Employee]
    val departmentDS = departmentDF.as[Department]

    employeeDS.joinWith(departmentDS, $"depId" === $"id").show()
  }
}