大数据开发之Spark篇----使用Dataset中遇到的小坑

最新推荐文章于 2024-03-04 16:58:19 发布

豆豆总

最新推荐文章于 2024-03-04 16:58:19 发布

阅读量3.2k

点赞数 2

CC 4.0 BY-SA版权

分类专栏： Spark

本文链接：https://blog.youkuaiyun.com/weixin_39702831/article/details/88653646

Dataset的使用

按照官网上面的提示，我将DataFrame转换成Dataset来使用，代码如下：

package com.doudou.www.sql01

import org.apache.spark.sql.{Encoders, SparkSession}


object JDBCApp {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.
      builder().appName("JDBCApp").master("local[2]").enableHiveSupport().getOrCreate()


    val person = spark.read.format("jdbc")
      .option("url","jdbc:mysql://doudou:3306/test")
      .option("user","doudou")
      .option("password","123456")
      .option("dbtable","person").load()

    person.createOrReplaceTempView("person")
    val df = spark.sql("select name,age,gender,case when gender='male' then '男' else '女' end as sex from person")

    import spark.implicits._

    case class Person(name:String,age:Int,gender:String)

    val df2 = df.select($"name",$"age",$"gender")
    val df3 = df2.as[Person]
//    val df2 = df.select("name,age,gender").as[Person]
    df3.filter(x => x.name == "doudou" |