SparkSQL 整合Hive

最新推荐文章于 2025-04-21 17:24:45 发布

Joker_Jiang3

最新推荐文章于 2025-04-21 17:24:45 发布

阅读量319

点赞数

分类专栏：大数据文章标签： spark 大数据

本文链接：https://blog.youkuaiyun.com/Joker_Jiang3/article/details/118567504

版权

SparkSQL 整合Hive

一、SparkSQL命令行界面整合Hive

1、先启动Hive的metastore

安装hive的目录下执行：

bin/hive --server metastore &

2、把hive的配置文件hive-site.xml拷贝到spark的conf

3、启动spark bin目录下的spark-sql命令

4、执行sql语句

二、SparkSQL代码整合Hive

完整的pom依赖：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.jiang</groupId>
    <artifactId>spark</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.12.10</scala.version>
        <hadoop.version>3.2.0</hadoop.version>
        <spark.version>3.0.1</spark.version>
    </properties>

    <dependencies>

        <!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.12.10</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
        <dependency>
            <groupId>org.apache.spark</groupId