本文详细介绍如何在Windows环境下配置Hadoop与Hive的集成，包括安装配置JDK、Scala、Hadoop等步骤，以及如何解决权限问题。此外，还提供了pom.xml依赖配置和hive-site.xml的具体设置。
| NO | 步骤 |
| 1 | 安装配置jdk1.8 |
| 2 | 安装配置scala2.11.18 |
| 3 | 写操作hive的demo |
| 4 | 导入pom.xml中依赖jar |
| 5 | 下载hadoop的binary包，我的版本是2.7.3 |
| 6 | 下载winutils.exe将其放到HADOOP_HOME/bin/目录下</td></tr><tr><td>7</td><td>在启动类的运行参数中设置环境变量，HADOOP_HOME=D:\software1\hadoop-2.7.3，=后面是hadoop的安装目录</td></tr><tr><td>8</td><td>将hive-site.xml配置文件放到工程的resource目录中</td></tr><tr><td>9</td><td>如果有权限问题，可以关闭namenode权限检查</td></tr></tbody></table><p>pom.xml的依赖关系如下：</p><pre οnclick="hljs.copyCode(event)"><code class="language-html hljs xml"><ol class="hljs-ln" style="width:918px"><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="1"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="php"><span class="hljs-meta"><?</span>xml version=<span class="hljs-string">"1.0"</span> encoding=<span class="hljs-string">"UTF-8"</span><span class="hljs-meta">?></span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="2"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-tag"><<span class="hljs-name">project</span> <span class="hljs-attr">xmlns</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0"</span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="3"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xmlns:xsi</span>=<span class="hljs-string">"http://www.w3.org/2001/XMLSchema-instance"</span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="4"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xsi:schemaLocation</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="5"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">modelVersion</span>></span>4.0.0<span class="hljs-tag"></<span class="hljs-name">modelVersion</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="6"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="7"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>com.lwq<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="8"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>sparkcase2<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="9"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>1.0-SNAPSHOT<span class="hljs-tag"></<span class="hljs-name">version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="10"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="11"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="12"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">java.version</span>></span>1.8<span class="hljs-tag"></<span class="hljs-name">java.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="13"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">spark.version</span>></span>2.1.1<span class="hljs-tag"></<span class="hljs-name">spark.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="14"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"></<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="15"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="16"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependencies</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="17"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependency</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="18"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.spark<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="19"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>spark-core_2.11<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="20"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>HADOOP_HOME/bin/目录下</td></tr><tr><td>7</td><td>在启动类的运行参数中设置环境变量，HADOOP_HOME=D:\software1\hadoop-2.7.3，=后面是hadoop的安装目录</td></tr><tr><td>8</td><td>将hive-site.xml配置文件放到工程的resource目录中</td></tr><tr><td>9</td><td>如果有权限问题，可以关闭namenode权限检查</td></tr></tbody></table><p>pom.xml的依赖关系如下：</p><pre οnclick="hljs.copyCode(event)"><code class="language-html hljs xml"><ol class="hljs-ln" style="width:918px"><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="1"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="php"><span class="hljs-meta"><?</span>xml version=<span class="hljs-string">"1.0"</span> encoding=<span class="hljs-string">"UTF-8"</span><span class="hljs-meta">?></span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="2"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-tag"><<span class="hljs-name">project</span> <span class="hljs-attr">xmlns</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0"</span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="3"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xmlns:xsi</span>=<span class="hljs-string">"http://www.w3.org/2001/XMLSchema-instance"</span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="4"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xsi:schemaLocation</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="5"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">modelVersion</span>></span>4.0.0<span class="hljs-tag"></<span class="hljs-name">modelVersion</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="6"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="7"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>com.lwq<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="8"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>sparkcase2<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="9"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>1.0-SNAPSHOT<span class="hljs-tag"></<span class="hljs-name">version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="10"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="11"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="12"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">java.version</span>></span>1.8<span class="hljs-tag"></<span class="hljs-name">java.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="13"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">spark.version</span>></span>2.1.1<span class="hljs-tag"></<span class="hljs-name">spark.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="14"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"></<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="15"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="16"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependencies</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="17"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependency</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="18"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.spark<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="19"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>spark-core_2.11<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="20"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>{spark.version}</version> |
- </dependency>
- <dependency>
- <groupId>mysql</groupId>
- <artifactId>mysql-connector-java</artifactId>
- <version>5.1.38</version>
- </dependency>
- <dependency>
- <groupId>org.apache.spark</groupId>
- <artifactId>spark-hive_2.11</artifactId>
- <version>${spark.version}</version>
- </dependency>
- </dependencies>
- </project>
- hive-site.xml的配置如下:
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <configuration>
- # hive元数据服务的地址
- <property>
- <name>hive.metastore.uris</name>
- <value>thrift://master:9083</value>
- </property>
- <property>
- <name>hive.server2.thrift.port</name>
- <value>10000</value>
- </property>
- <property> #元数据地址
- <name>javax.jdo.option.ConnectionURL</name>
- <value>jdbc:mysql://master/metastore?createDatabaseIfNotExist=true</value>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionDriverName</name>
- <value>com.mysql.jdbc.Driver</value>
- </property>
- <property> #连接元数据数据库账号
- <name>javax.jdo.option.ConnectionUserName</name>
- <value>hadoop</value>
- </property>
- <property> #连接元数据数据库密码
- <name>javax.jdo.option.ConnectionPassword</name>
- <value>hadoop</value>
- </property>
- <property> #hive数据存储在hdfs上的目录
- <name>hive.metastore.warehouse.dir</name>
- <value>/warehouse</value>
- </property>
- <property>
- <name>fs.defaultFS</name>
- <value>hdfs://master:9000</value>
- </property>
- <property> #元数据schema验证
- <name>hive.metastore.schema.verification</name>
- <value>false</value>
- </property>
- <property>
- <name>datanucleus.autoCreateSchema</name>
- <value>true</value>
- </property>
- <property>
- <name>datanucleus.autoStartMechanism</name>
- <value>SchemaTable</value>
- </property>
- <property>
- <name>datanucleus.schema.autoCreateTables</name>
- <value>true</value>
- </property>
- <property>
- <name>beeline.hs2.connection.user</name>
- <value>bigdata</value>
- </property>
- <property>
- <name>beeline.hs2.connection.password</name>
- <value>root</value>
- </property>
- </configuration>
演示代码:
- package com.lwq.spark
- import org.apache.spark.sql.SparkSession
- object HiveCaseJob {
- def main(args: Array[String]): Unit = {
- val sparkSession = SparkSession.builder().appName("HiveCaseJob").master("local[*]").enableHiveSupport().getOrCreate()
- sparkSession.sql("drop table if exists users")
- sparkSession.sql("show tables").show()
- sparkSession.sql("create table if not exists users(id int,name string) row format delimited fields terminated by ' ' stored as textfile")
- sparkSession.sql("show tables").show()
- sparkSession.sql("select * from users").show()
- sparkSession.sql("load data local inpath 'src/main/resources/a.txt' overwrite into table users")
- sparkSession.sql("select * from users").show()
- }
- }
配置运行参数:
运行过程中可能的报错：
这是HADOOP_HOME变量和winutils.exe的原因，查看变量是否配好，winutils是否放在正确目录。
因为是在windows环境，在运行的时候会模拟linux，需要winutils.exe，winutils.exe的路径代码中默认是从hadoop的安装路径的bin目录下。所以还需要上面的HADOOP_HOME变量。
还有一个错误就是提示权限不足的，因为在windows下是通过windows的用户来访问的，可以关闭hdfs的权限检查
如下：
扫一扫
2万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
