Spark 连接hive local

最新推荐文章于 2023-11-07 17:41:10 发布

原创最新推荐文章于 2023-11-07 17:41:10 发布 · 1.9k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#hive #spark #大数据

本文介绍了两种方式通过Spark连接Hive：一是使用HiveServer2服务，二是通过MetaStore服务的Thrift协议。在配置文件hive-site.xml中设置了数据库连接参数，并展示了尝试连接过程的日志信息。在HiveServer2方式下成功建立连接，而在MetaStore服务方式下出现连接失败的警告。

采用读取recources文件夹下的配制文件方式hive-site.xml 
一.通过hiveserver2 服务连接     $HIVE_HOME/bin/hiveserver2
object NewSparkHiveDemo {

    def main(args: Array[String]): Unit = {

        System.setProperty("HADOOP_USER_NAME", "fengjiangyue")

        val spark: SparkSession = SparkSession.builder()
          .appName("spark hive Example")
          .master("local")
//          .config("hive.metastore.uris", "thrift://localhost:9083")
          .config("spark.sql.warehouse.dir", "hdfs://localhost:9000/user/hive/warehouse/")
          .enableHiveSupport()
          .getOrCreate()

        import spark.implicits._
        import spark.sql

        sql("show tables").show()

//        sql("create table if not exists src(key INT, value STRING)")
//        val data = Array((1, "val1"),(2, "val2"),(3,"val3"))
//        var df = spark.createDataFrame(data).toDF("key","value")
//        df.createOrReplaceTempView("temp_src")
//        sql("insert into src select key,value from temp_src")
//        sql("select * from src").show()

    }
}

hive-site.xml如下

<configuration>
   <property>
     <name>javax.jdo.option.ConnectionURL</name>
     <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
     <description>JDBC connect string for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionDriverName</name>
     <value>com.mysql.jdbc.Driver</value>
     <description>Driver class name for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionUserName</name>
     <value>root</value>
     <description>username to use against metastore database</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionPassword</name>
     <value>root</value>
     <description>password to use against metastore database</description>
   </property>
</configuration>

21/12/25 08:18:08 INFO HiveConf: Found configuration file file:/Users/fengjiangyue/IdeaProjects/sparklearn/sparkdoit/target/classes/hive-site.xml
21/12/25 08:18:09 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/0266e12f-79bc-4c58-bad7-be9bbeebb47b
21/12/25 08:18:09 INFO SessionState: Created local directory: /var/folders/f8/y6vdl3zs5_10j0tyq235py540000gn/T/fengjiangyue/0266e12f-79bc-4c58-bad7-be9bbeebb47b
21/12/25 08:18:09 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/0266e12f-79bc-4c58-bad7-be9bbeebb47b/_tmp_space.db
21/12/25 08:18:09 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is hdfs://localhost:9000/user/hive/warehouse/
21/12/25 08:18:10 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/12/25 08:18:10 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/12/25 08:18:10 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
21/12/25 08:18:10 INFO ObjectStore: ObjectStore, initialize called
21/12/25 08:18:10 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/12/25 08:18:10 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/12/25 08:18:11 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/12/25 08:18:12 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
21/12/25 08:18:12 INFO ObjectStore: Initialized ObjectStore
21/12/25 08:18:12 INFO HiveMetaStore: Added admin role in metastore
21/12/25 08:18:12 INFO HiveMetaStore: Added public role in metastore
21/12/25 08:18:12 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/12/25 08:18:12 INFO HiveMetaStore: 0: get_all_functions
21/12/25 08:18:12 INFO audit: ugi=fengjiangyue ip=unknown-ip-addr cmd=get_all_functions 21/12/25 08:18:12 INFO HiveMetaStore: 0: get_database: default

二.通过metastore 服务的thrift协义开启$HIVE_HOME/bin/hive --service hiveserver2

object NewSparkHiveDemo {
    def main(args: Array[String]): Unit = {
        System.setProperty("HADOOP_USER_NAME", "fengjiangyue")
        val spark: SparkSession = SparkSession.builder()
          .appName("spark hive Example")
          .master("local")
          .config("hive.metastore.uris", "thrift://localhost:9083")
          .config("spark.sql.warehouse.dir", "hdfs://localhost:9000/user/hive/warehouse/")
          .enableHiveSupport()
          .getOrCreate()

        import spark.implicits._
        import spark.sql

        sql("show tables").show()
    }
}

hive-site.xml配制如下：
<configuration>
    <property>
       <name>hive.metastore.uris</name>
       <value>thrift://localhost:9083</value>
   </property>
</configuration>

21/12/25 08:15:27 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
21/12/25 08:15:27 INFO HiveConf: Found configuration file file:/Users/fengjiangyue/IdeaProjects/sparklearn/sparkdoit/target/classes/hive-site.xml
21/12/25 08:15:28 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/8abeee58-2369-4a59-aa80-f341c182a0e8
21/12/25 08:15:28 INFO SessionState: Created local directory: /var/folders/f8/y6vdl3zs5_10j0tyq235py540000gn/T/fengjiangyue/8abeee58-2369-4a59-aa80-f341c182a0e8
21/12/25 08:15:28 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/8abeee58-2369-4a59-aa80-f341c182a0e8/_tmp_space.db
21/12/25 08:15:28 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is hdfs://localhost:9000/user/hive/warehouse/
21/12/25 08:15:28 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083
21/12/25 08:15:28 WARN metastore: Failed to connect to the MetaStore Server...
21/12/25 08:15:28 INFO metastore: Waiting 1 seconds before next connection attempt.
21/12/25 08:15:29 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083
21/12/25 08:15:29 WARN metastore: Failed to connect to the MetaStore Server...
21/12/25 08:15:29 INFO metastore: Waiting 1 seconds before next connection attempt.
21/12/25 08:15:30 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083
21/12/25 08:15:30 WARN metastore: Failed to connect to the MetaStore Server...
21/12/25 08:15:30 INFO metastore: Waiting 1 seconds before next connection attempt.
21/12/25 08:15:31 WARN Hive: Failed to register all functions.