SparkSQL对mysql的读写

关注

原创最新推荐文章于 2026-03-18 00:38:09 发布 703 阅读 AI 写同款· GEO 优化 ›

sparksql jdbc mysql_SparkSQL-通过JDBC读写mysql数据库 object JdbcDatasourceTest {def main(args: Array[String]): Unit = {val spark = SparkSession.builder().appName("JdbcDatasourceTest").master("local").getOrCreate()//url:// jdbc:mysql://master:3306/test//... 阅读详情

SparkSQL JDBC数据源

！：load这个方法不会真正读取mysql的数据

//1、创建sparkSession
val sparkSession: SparkSession = SparkSession.builder().appName("JdbcDataSource").master("local[*]").getOrCreate()
    import sparkSession.implicits._
//2、从数据库中读取数据
    val logs: DataFrame = sparkSession.read.format("jdbc").options(
      Map("url" -> "jdbc:mysql://localhost:3306/bigdata",
        "driver" -> "com.mysql.jdbc.Driver",
        "dbtable" -> "log",
        "user" -> "root",
        "password" -> ""
      )
    ).load()

    //logs.printSchema()输出schema信息
	
    //logs.filter($"age" >12) 过滤年龄大于12的人，查询结果包括了12
	//方式二，不包含12
/*    val filtered:Dataset[Row] = logs.filter(
      r => {
        r.getAs[Int](2) < 12
      }
    )
    filtered.show()*/

	
    val nlog: DataFrame = logs.select($"id",$"name",$"age"*10 as "new_age")
	//将数据写到mysql中
    val pope = new Properties()
    val props = new Properties()
    props.put("user","root")
    props.put("password","")
    nlog.write.mode("ignore").jdbc("jdbc:mysql://localhost:3306/bigdata", "logs1", props)
    logs.show()
    sparkSession.close()
    ```

收录于