hive on spark 代码方式实现

最新推荐文章于 2024-12-17 19:50:53 发布

IT_ZhiCunGaoYuan

最新推荐文章于 2024-12-17 19:50:53 发布

阅读量401

点赞数

CC 4.0 BY-SA版权

分类专栏：大数据文章标签： hive spark

本文链接：https://blog.youkuaiyun.com/IT_liuzhiyuan/article/details/125526942

大数据专栏收录该内容

16 篇文章

订阅专栏

本文档提供了一个CDH6.3集群中Spark 2.4.0-cdh6.3.2、Hadoop 3.0.0-cdh6.3.2和Hive 2.1.1-cdh6.3.2的版本配置示例。展示了如何在Maven项目中配置相关依赖，并给出了SparkSession连接Hive的代码实现。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

CDH 版本 6.3

    spark 版本  2.4.0-cdh6.3.2
    hadoop 版本 3.0.0-cdh6.3.2
    hive 版本 2.1.1-cdh6.3.2

pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>bigdata</groupId>
    <artifactId>data_shuffle_demo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>

        <scala.version>2.11.12</scala.version>
        <spark.version>2.4.0-cdh6.3.2</spark.version>
        <hadoop.version>3.0.0-cdh6.3.2</hadoop.version>
        <!--项目打包: provided  项目本地运行调试: compile-->
        <mvn.scop.pro>provided</mvn.scop.pro>
        <!-- 加了provided的依赖要保证在spark服务器的sparkJar目录里存在, 具体路径/opt/cloudera/parcels/CDH/lib/spark/jars -->
        <!--jar包前缀-->
        <jar.pre.name>optimization</jar.pre.name>
    </properties>

    <repositories>

        <repository>

            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>

    </repositories>

        <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>${mvn.scop.pro}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <scope>${mvn.scop.pro}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
            <scope>${mvn.scop.pro}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>${mvn.scop.pro}</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>2.1.1-cdh6.3.2</version>
        </dependency>

        <dependency>
        <groupId>org.apache.hive</groupId>
        <artifactId>hive-jdbc</artifactId>
        <version>2.1.1-cdh6.3.2</version>
        <scope>${mvn.scop.pro}</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>${mvn.scop.pro}</scope>
        </dependency>

    </dependencies>

    <build>
        <finalName>${jar.pre.name}</finalName>
        <plugins>


            <!--设置编译版本-->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>


            <!--声明并引入构建插件-->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>


        </plugins>
    </build>

</project>

sparksession 连接hive

//spark链接
  def sparkSession(): SparkSession = {

    var conf = new SparkConf()
      .set("spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation", "true")
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .set("spark.sql.warehouse.dir", "your hive warehouse")
      .set("hive.metastore.uris", "your hive metastore uris")
    SparkSession.builder()
      .config(conf)
      .enableHiveSupport()
      .getOrCreate()
  }