intergrate sparkSQL with hive

最新推荐文章于 2024-08-08 17:31:50 发布

原创最新推荐文章于 2024-08-08 17:31:50 发布 · 467 阅读

0 ·

CC 4.0 BY-SA版权

大数据专栏收录该内容

29 篇文章

订阅专栏

本文介绍如何配置Hive以使用MySQL作为元数据存储，并解决启动Spark-shell时出现的相关异常问题。通过调整Hive配置文件及添加MySQL连接驱动，确保Hive与Spark能正确连接并读取元数据。

1.configuration

copy hive-site to {spack_HOME}/conf ,and add two properties:

    <property>
       <name>hive.metastore.schema.verification</name>
       <value>false</value>
        <description>
        Enforce metastore schema version consistency.
        True: Verify that version information stored in metastore matches with one from Hive jars. Also disable automatic
              schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures
              proper metastore schema migration. (Default)
        False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
        </description>
     </property>

2.细节问题

默认情况下，Hive是使用derby保存metadata元数据，若使用Mysql来保存metadata，在启动Spark-shell是会出现一个异常

Exception in thread "main" Java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.Hadoop.hive.metastore.HiveMetaStoreClient

解决的方法是添加MySQL的连接驱动到

[plain] view plain copy