centos7 Hive2.3.6安装与配置

一.简介

• Hive 由 Facebook 实现并开源，是基于 Hadoop 的一个数据仓库工具，可以将结构化的数据映射为一张数据库表，并提供 HQL(Hive SQL)查询功能，底层数据是存储在 HDFS 上。
• Hive 本质: 将 SQL 语句转换为 MapReduce 任务运行，使不熟悉 MapReduce 的用户很方便地利用 HQL 处理和计算 HDFS 上的结构化的数据,是一款基于 HDFS 的 MapReduce 计算框架
主要用途：用来做离线数据分析，比直接用 MapReduce 开发效率更高。

二.安装与配置

基于<<Hadoop2.7.7 HA完全分布式集群搭建>>

1.下载 Hive2.3.6

cd /usr/local
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz

tar -zxvf apache-hive-2.3.6-bin.tar.gz

2.配置hive环境变量

# 编辑 /etc/profile 文件, 在其中添加以下内容 
vim /etc/profile
export HIVE_HOME=/usr/local/apache-hive-2.3.6-bin
export PATH=$PATH:$HIVE_HOME/bin

# 激活配置
source /etc/profile

3.在hadoop下创建hive所用文件夹

$HADOOP_HOME/bin/hadoop fs -mkdir  -p     /usr/local/apache-hive-2.3.6/tmp
$HADOOP_HOME/bin/hadoop fs -mkdir  -p     /usr/local/apache-hive-2.3.6/warehouse
$HADOOP_HOME/bin/hadoop fs -chmod g+w   /usr/local/apache-hive-2.3.6/tmp
$HADOOP_HOME/bin/hadoop fs -chmod g+w   /usr/local/apache-hive-2.3.6/warehouse

4.下载对应的mysql连接jar

https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-versions.html
https://dev.mysql.com/downloads/connector/j/

将目标文件移动到 hive的lib文件夹下

mv mysql-connector-java-5.1.48.jar $HIVE_HOME/lib

创建数据库,配置用户和权限

create database metastore;
grant all on metastore.* to hive@'%'  identified by 'hive';
grant all on metastore.* to hive@'localhost'  identified by 'hive';
flush privileges;

5.配置 Hive

cd apache-hive-2.3.6-bin/conf/
cp hive-env.sh.template hive-env.sh 
cp hive-default.xml.template hive-site.xml 
cp hive-log4j2.properties.template hive-log4j2.properties 
cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties

修改 vim hive-env.sh

#Java路径
export JAVA_HOME=/usr/local/java
#Hadoop安装路径
export HADOOP_HOME=/usr/local/hadoop-2.7.7
#Hive安装路径
export HIVE_HOME=/usr/local/apache-hive-2.3.6-bin
#Hive配置文件路径
export HIVE_CONF_DIR=/usr/local/apache-hive-2.3.6-bin/conf
# hive存放外部jar包的位置
export HIVE_AUX_JARS_PATH=/usr/local/apache-hive-2.3.6-bin/bin

修改hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>

<!-- Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI -->
<property>
    <name>hive.exec.scratchdir</name>
    <value>/usr/local/apache-hive-2.3.6/tmp</value>
</property>
<property>
    <name>hive.exec.local.scratchdir</name>
   
    <!-- Relative path in absolute URI: ${system:user.name%7D -->
    <!--  去掉system: -->
    <value>/usr/local/apache-hive-2.3.6/tmp/${user.name}</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>/usr/local/apache-hive-2.3.6/tmp/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>

<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/usr/local/apache-hive-2.3.6/warehouse</value>
</property>
<property>
    <name>hive.querylog.location</name>
    <value>/usr/local/apache-hive-2.3.6/logs/${user.name}</value>
</property>
 
<property>
  <name>hive.server2.logging.operation.log.location</name>
  <!-- Relative path in absolute URI: ${system:user.name%7D -->
    <!--  去掉system: -->
  <value>/usr/local/apache-hive-2.3.6/logs/${user.name}/operation_logs</value>
  <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
</property>

<!-- 配置 MySQL 数据库连接信息 -->
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8&amp;useSSL=false</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value><!-- 指定mysql用户名 -->
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value><!-- 指定mysql密码 -->
  </property>
</configuration>

6.初始化hive

cd $HIVE_HOME/bin
./schematool -dbType mysql -initSchema hive hive

7.启动 hive的metastore元数据服务

hive --service metastore

8.启动hive

hive

Exception in thread "main" org.apache.thrift.transport.TTransportException:
# 查看端口情况
lsof -i:9083 
# 再关闭端口即可
kill -9 pid

参考:https://blog.youkuaiyun.com/wodedipang_/article/details/72718138
参考:https://blog.youkuaiyun.com/baixf/article/details/101052279
参考:https://www.cnblogs.com/xibuhaohao/p/11772481.html