hive0.13整合hbase 0.96.2 hadoop2.2.0

最新推荐文章于 2021-11-12 18:26:57 发布

最新推荐文章于 2021-11-12 18:26:57 发布 · 182 阅读

文章标签：

#hive0.13 整合 hbase0.96.2

hive整合hbase 专栏收录该内容

1 篇文章

订阅专栏

本文介绍如何通过配置Hive连接HBase实现数据交互，包括必要的Hive配置及依赖库上传步骤，并解决了一个常见的文件未找到错误。

最近研究了一下hive与hbase的整合，使用的都是各自的最新release， hive0.13, hbase0.96.2，整合的过程其实挺简单的，大致需要注意的地方如下：

1. hive的配置文件hive-site.xml需要添加的内容：

<property>
<name>hive.aux.jars.path</name>
<value>file:///home/grid/hive/lib/hive-hbase-handler-0.13.0.jar,file:///home/grid/hive/lib/hbase-client-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-common-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-common-0.96.2-hadoop2-tests.jar,file:///home/grid/hive/lib/hbase-protocol-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-server-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/htrace-core-2.04.jar,file:///home/grid/hive/lib/zookeeper-3.4.6.jar,file:///home/grid/hive/lib/protobuf-java-2.5.0.jar,file:///home/grid/hive/lib/guava-11.0.2.jar</value>
</property>

<property>
<name>hbase.zookeeper.quorum</name>
<value>server1,server2</value>
</property>

2.安装hbase的过程不再累述，可以参考http://blog.youkuaiyun.com/codestinity/article/details/6947464

最后在双方查询都能查到数据，但是在hive中向hbase插入数据的时候出现了问题，错误信息如下：

java.io.FileNotFoundException: File does not exist: hdfs://*.*.*.*:9000/home/grid/hbase/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:99)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://*.*.*.*:9000/home/grid/hbase/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar)'
Execution failed with exit status: 1
Obtaining error information