hadoop学习笔记-hive安装及操作

软件下载：

Mysql:

http://mysql.llarian.net/Downloads/MySQL-5.5/MySQL-5.5.24-1.rhel5.x86_64.tar

Hive:

http://apache.dataguru.cn/hive/hive-0.8.1/hive-0.8.1.tar.gz

 

安装环境：

OS:Oracle redhad 5.6 x86 64bit

Hadoop: hadoop-0.20.2

Mysql:mysql-5.5.24

Hive:hive-0.8.1

 

1.       安装mysql

--解压

[root@gc ~]# tar xvf MySQL-5.5.24-1.rhel5.x86_64.tar

MySQL-client-5.5.24-1.rhel5.x86_64.rpm

MySQL-devel-5.5.24-1.rhel5.x86_64.rpm

MySQL-embedded-5.5.24-1.rhel5.x86_64.rpm

MySQL-server-5.5.24-1.rhel5.x86_64.rpm

MySQL-shared-5.5.24-1.rhel5.x86_64.rpm

MySQL-test-5.5.24-1.rhel5.x86_64.rpm

 

--安装

[root@gc mysql]# rpm -ivh *

Preparing...                ########################################### [100%]

   1:MySQL-client           ########################################### [ 17%]

   2:MySQL-devel            ########################################### [ 33%]

   3:MySQL-embedded         ########################################### [ 50%]

   4:MySQL-server           ########################################### [ 67%]

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !

To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'

/usr/bin/mysqladmin -u root -h gc.localdomain password 'new-password'

Alternatively you can run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test

databases and anonymous user created by default.  This is

strongly recommended for production servers.

See the manual for more instructions.

Please report any problems with the /usr/bin/mysqlbug script!

   5:MySQL-shared           ########################################### [ 83%]

   6:MySQL-test             ########################################### [100%]

 

 

--创建数据库hive

mysql> create database hive;

 

--创建hive用户,并授权

mysql> grant all on hive.* to hive@'%' identified by 'hive';

mysql> grant all on hive.* to hive@' gc.localdomain' identified by 'hive';

mysql> grant all on hive.* to hive@' localhost' identified by 'hive';

mysql> flush privileges;

 

2.       安装hive

--解压

[grid@gc ~]$ pwd

/home/grid

[grid@gc ~]$ tar xzvf hive-0.8.1.tar.gz

 

--添加环境量

export HIVE_HOME=/home/grid/hive-0.8.1

export PATH=$HIVE_HOME/bin:$ PATH

 

--修改hive-site.xml

<property>

  <name>javax.jdo.option.ConnectionURL</name>

  <value>jdbc:mysql://gc:3306/hive?createDatabaseIfNotExist=true</value>

  <description>JDBC connect string for a JDBC metastore</description>

</property>

 

<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>com.mysql.jdbc.Driver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

 

<property>

  <name>javax.jdo.option.ConnectionUserName</name>

  <value>hive</value>

  <description>username to use against metastore database</description>

</property>

 

<property>

  <name>javax.jdo.option.ConnectionPassword</name>

  <value>hive</value>

  <description>password to use against metastore database</description>

</property>

 

<property>

  <name>hive.metastore.local</name>

  <value>true</value>

  <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>

</property>

 

<property>

  <name>hive.metastore.warehouse.dir</name>

  <value>/home/grid/hive/warehouse</value>

  <description>location of default database for the warehouse</description>

</property>

 

<property>

  <name>hive.exec.scratchdir</name>

  <value>/tmp/hive-${user.name}</value>

  <description>Scratch space for Hive jobs</description>

</property>

 

<property>

  <name>hive.metastore.uris</name>

  <value>thrift://127.0.0.1:9083</value>

</property>

 

--复制mysql-connector-java-x.x.x.jar文件到hive的lib目录下

[root@gc ~]# find / -name "mysql-connector-java*" -print

/home/oracle/Oracle/Middleware/wlserver_10.3/server/lib/mysql-connector-java-commercial-5.0.3-bin.jar

/home/oracle/Oracle/Middleware/wlserver_10.3/server/ext/jdbc/mysql/mysql-connector-java-commercial-5.0.3-bin.jar

----在我之前安装的weblogic目录下找到了此文件

----copy到hive的lib目录

[root@gc lib]# cp mysql-connector-java-commercial-5.0.3-bin.jar /home/grid/hive-0.8.1/lib

 

 

--登录hive

[grid@gc conf]$ hive

Logging initialized using configuration in jar:file:/home/grid/hive-0.8.1/lib/hive-common-0.8.1.jar!/hive-log4j.properties

Hive history file=/tmp/grid/hive_job_log_grid_201301180705_1729707460.txt

hive>

 

 

3.       hive操作

测试语句

•建立测试表test

•create table test (key string);

•show tables;

 

 

4.       遇到的问题

1)        .

现象：

[grid@gc conf]$ hive

Logging initialized using configuration in jar:file:/home/grid/hive-0.8.1/lib/hive-common-0.8.1.jar!/hive-log4j.properties

Hive history file=/tmp/grid/hive_job_log_grid_201301180637_1509680201.txt

hive> create table test (key string);

FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory

NestedThrowables:

java.lang.reflect.InvocationTargetException

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

 

Hive日志

Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

 

解决：

复制mysql-connector-java-commercial-5.0.3-bin.jar文件到/home/grid/hive-0.8.1/lib目录

 

参考：

 

 

2)         

现象：

hive> show tables;

FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Access denied for user 'hive'@'gc.localdomain' (using password: YES)

NestedThrowables:

java.sql.SQLException: Access denied for user 'hive'@'gc.localdomain' (using password: YES)

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

[grid@gc conf]$ mysql -uhive -phive

ERROR 1045 (28000): Access denied for user 'hive'@'localhost' (using password: YES)

 

解决：

mysql> grant all on hive.* to hive@' gc.localdomain' identified by 'hive';

mysql> grant all on hive.* to hive@' localhost' identified by 'hive';

 

参考文章：

http://f.dataguru.cn/blog-303-610.html

http://blog.youkuaiyun.com/lichangzai/article/details/8230988

 

hive-0.11.0安装配置可以参考下面的文章

http://f.dataguru.cn/forum.php?mod=viewthread&tid=197019

 

在配置安装hive-0.11.0时，可能遇到下面的错：

[licz@gc bin]$ hive
Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

 

解决办法：

[licz@nticket1 ~]$ cd ~/hive-0.11.0/conf/

[licz@nticket1 ~]$ cp hive-env.sh.template hive-env.sh

--在hive-env.sh文件里加下面的内容：

[licz@gc conf]$ vi hive-env.sh
export HADOOP_HOME=/home/licz/hadoop-1.2.1/                                                                                                                   
[licz@gc conf]$ source hive-env.sh

3)  hive-site.xml配置文件位置

在hive-0.11.0版本以后，和0.8版本不同的是，hive-site.xml配置文默认是不在 ~/hive-0.11.0/conf/目录下

这时可以重新生成一个，如下：

[licz@nticket1 ~]$ cd ~/hive-0.11.0/conf/
[licz@nticket1 conf]$ ls
hive-default.xml.template  hive-env.sh.template  hive-exec-log4j.properties.template  hive-log4j.properties.template

[licz@nticket1 conf]$ cp hive-default.xml.template hive-site.xml

--修改配置信息（和0.8版本相同配置）
[licz@nticket1 ~]$ vi ~/hive-0.11.0/conf/hive-site.xml