Hadoop3.3安装配置

原创已于 2022-04-20 16:48:32 修改 · 2k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#java #大数据

于 2022-04-19 12:35:30 首次发布

Hadoop 专栏收录该内容

33 篇文章

订阅专栏

本文档详细介绍了在MacOS上安装和配置Hadoop 3.3和Hive 3.1.3的步骤，包括使用Homebrew安装，设置环境变量，配置Hadoop和Hive的相关文件如core-site.xml和hdfs-site.xml，以及初始化元数据和启动服务。同时，还涉及到MySQL JDBC驱动的添加，HDFS目录的创建和权限设置，以及通过Beeline连接Hive的步骤。

1. macOS安装

brew install hadoop
brew install hive

Hadoop3.3安装

2. 参数配置

参考地址：Hive安装配置
(1)环境变量：

## java
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

## go
export GOROOT=/usr/local/Cellar/go/1.18
export GOPATH=/Users/apple/workspace/GoProjects
export GOBIN=$GOPATH/bin
export GO111MODULE=on
export GOPROXY=https://goproxy.cn,https://goproxy.io,direct
export GOSUMDB=sum.golang.google.cn

## hadoop
export HADOOP_HOME=/usr/local/Cellar/hadoop/3.3.2/libexec
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CLASSPATH=`hadoop classpath`
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native/"

## hive
export HIVE_HOME=/usr/local/Cellar/hive/3.1.3/libexec
export HIVE_CONF_DIR=$HIVE_HOME/conf

(2)hadoop-env.sh

#############################################################
export JAVA_HOME=/usr/local/Cellar/openjdk@8/1.8.0+322
export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/3.3.2/libexec/etc/hadoop

(3)core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
	<!-- 指定HDFS中NameNode的地址 -->
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://192.168.31.120:9000</value>
	</property>

	<!-- 指定Hadoop运行时产生的文件的存储目录 -->
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/usr/local/Cellar/hadoop/3.3.2/data/tmp</value>
	</property>

</configuration>

(4)hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
	<!-- 指定HDFS副本上数量 -->
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>

	<!-- 可通过关闭权限检 -->
	<property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
	</property>
</configuration>

3. Hive配置

(1)hive-env.sh

#hadoop
HADOOP_HOME=/usr/local/Cellar/hadoop/3.3.2/libexec
#hive conf
export HIVE_CONF_DIR=/usr/local/Cellar/hive/3.1.3/libexec/conf
#hive lib
export HIVE_AUX_JARS_PATH=/usr/local/Cellar/hive/3.1.3/libexec/lib

(2)hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!-- jdbc连接的URL -->
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;useSSL=false&amp;serverTimezone=GMT</value>
	</property>

    <!-- jdbc连接的Driver-->
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.cj.jdbc.Driver</value>
	</property>

    <!-- jdbc连接的username-->
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>

    <!-- jdbc连接的password -->
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456aA</value>
    </property>

    <property>
        <name>system:java.io.tmpdir</name>
        <value>/usr/local/Cellar/hive/3.1.3/data/tmp</value>
    </property>
    
   <property>
        <name>hive.cli.print.header</name>
        <value>true</value>
    </property>

    <property>
        <name>hive.cli.print.current.db</name>
        <value>true</value>
    </property>
</configuration>

(3)将MySQL的JDBC驱动拷贝到Hive的lib目录下:
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-8.0.27.tar.gz

(4)创建文件夹

##创建文件夹
hdfs dfs -mkdir -p /user/hive/warehouse
hadoop fs -mkdir -p /user/hive/tmp
hadoop fs -mkdir -p /user/hive/log

## 修改文件夹权限
hadoop fs -chmod -R 777 /user/hive/warehouse
hadoop fs -chmod -R 777 /user/hive/tmp
hadoop fs -chmod -R 777 /user/hive/log

(5)关闭安全模式
hdfs dfsadmin -safemode leave

(6)初始化元数据
schematool -dbType mysql -initSchema

(7).服务端后台开启metastore
nohup hive --service metastore &

(8)启动hiveserver2节点
nohup bin/hiveserver2 &

(9)连接

beeline> !connect jdbc:hive2://hadoop1:10000
#默认端口号是10000
Connecting to jdbc:hive2://hadoop1:10000
#输入账号密码
Enter username for jdbc:hive2://hadoop1:10000: root
Enter password for jdbc:hive2://hadoop1:10000: ******

(10)
hdfs: http://localhost:9870/
hive: http://localhost:10002/