spark 链接mysql/SQLServer数据库的pom文件依赖以及代码

最新推荐文章于 2024-06-17 14:25:05 发布

不会飞的乌龟

最新推荐文章于 2024-06-17 14:25:05 发布

阅读量1.6k

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/qq_42012160/article/details/88839847

spark 专栏收录该内容

10 篇文章

订阅专栏

本文提供使用Apache Spark连接并读取SQLServer及MySQL数据库的具体代码示例，并展示了如何配置pom文件来添加对应的数据库驱动依赖。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

pom文件依赖：

        <!--sqlserver数据库访问-->
        <dependency>
            <groupId>com.microsoft.sqlserver</groupId>
            <artifactId>mssql-jdbc</artifactId>
            <version>7.0.0.jre8</version>
            <!--<scope>provided</scope>-->
        </dependency>
        <!--mysql数据库访问-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.25</version>
           <!--<scope>provided</scope>-->
        </dependency>

链接SQLServer具体代码：

package com.lenovo.sqlserver

import org.apache.spark.sql.SparkSession

object ServerShipment {
  def main(args: Array[String]): Unit = {
    val ss = SparkSession
      .builder()
      .appName("ServerShipment")
      .master("yarn")
      //.config("spark.sql.warehouse.dir", "file:///D://lenovo_pj//cpp//cpp")
      //.enableHiveSupport()
      .getOrCreate()

    //sqlserver数据链接参数定义
    val driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    val url    = "jdbc:sqlserver://10.100.50.252:1433;DatabaseName=FF_GSC;" +
      "useUnicode=true&characterEncoding=UTF-8"
    val user   = "xxx"
    val pwd   = "xxx"
    val table  = "xxxxxx"

    //加载sqlserver数据库数据
    ss.read
      .format("jdbc")
      .option("driver",driver)
      .option("url",url)
      .option("user",user)
      .option("password",pwd)
      .option("dbtable",table)
      .load()
      .createOrReplaceTempView("Server_Shipment")
      .show()
      
    ss.stop()
  }
}

链接mysql具体代码：

package com.lenovo.tree
import org.apache.spark.sql.SparkSession
object LabelTreeRefresh {
  def main(args: Array[String]): Unit = {
    val ss = SparkSession
      .builder()
      .appName("LabelTreeRefresh")
      .master("local")
      //.config("spark.sql.warehouse.dir", "file:///D://lenovo_pj//cpp//cpp")
      .getOrCreate()
      
     val url      = "jdbc:mysql://10.96.91.186:3306/lenovo_channel_persona?      rewriteBatchedStatements=true&" +
                    "useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull"
     val driver   = "com.mysql.jdbc.Driver"
     val user     = "xxx"
     val password = "xxx"
    /**
      * 加载标签数据配置表数据(cpp_label_tree_refresh_config)
      */
    val refresh_config = "cpp_label_tree_refresh_config"
    ss.read
      .format("jdbc")
      .option("url",url).option("driver",driver)
      .option("user",user).option("password",password)
      .option("dbtable",refresh_config)
      .load()
      .createOrReplaceTempView("cpp_label_tree_refresh_config")
      .show()
      
    ss.stop()
  }
}