hadoop集群HA搭建（HADOOP配置 ubuntu18.04-live-server）

hadoop集群HA搭建-基础配置

- hadoop 集群HA搭建-基础配置

hadoop 集群HA搭建-基础配置

eg: 版本3.2.2
安装HA集群前请确认已安装zookeeper集群，配置SSH无密钥登陆

安装 java jdk

添加环境变量 sudo vim /etc/profile

# <<<<<< java start
export JAVA_HOME=/home/hadoop/jdk1.8.0_291
# >>>>>> java end
# <<<<<< hadoop start
export HADOOP_INSTALL=/home/hadoop/hadoop-3.2.2
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONFIG_DIR=$HADOOP_INSTALL/etc/hadoop
# >>>>>> hadoop end

添加java软链接到/bin/java

sudo ln -s /home/hadoop/jdk1.8.0_291/bin/java /bin/java

配置$HADOOP_HOME/etc/hadoop/hadoop-env.sh
```
JAVA_HOME=/home/hadoop/jdk1.8.0_291
```
使配置文件生效
```
sudo source /etc/profile
```

配置HADOOP配置文件

$HADOOP_HOME/etc/hadoop/core-site.xml

<!-- ha配置 -->
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://mycluster</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>file:/home/hadoop/hadoop-3.2.2/tmp</value>
  <description>Abase for other temporary directories.</description>
</property>
<!-- ha配置 -->
<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>file:/home/hadoop/hadoop-3.2.2/data/journalnode</value>
</property>
<property>
  <name>ha.zookeeper.quorum</name>
  <value>master:2181,node1:2181,node2:2181</value>
</property>

配置$HADOOP_HOME/etc/hadoop/hdfs-site.xml

<property>
  <name>dfs.replication</name>
  <value>2</value>
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/home/hadoop/hadoop-3.2.2/tmp/dfs/name</value>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/home/hadoop/hadoop-3.2.2/tmp/dfs/data</value>
</property>
<!-- ha配置 -->
<property>
  <name>dfs.permissions.enabled</name>
  <value>false</value>
</property>
<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>
<property>
  <name>dfs.nameservices</name>
  <value>mycluster</value>
</property>
<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>master,node1,node2</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.master</name>
  <value>master:8082</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.master</name>
  <value>master:9870</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.node1</name>
  <value>node1:8082</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.node1</name>
  <value>node1:9870</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.node2</name>
  <value>node2:8082</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.node2</name>
  <value>node2:9870</value>
</property>
<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node1:8485;node2:8485;master:8485/mycluster</value>
</property>
<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- zookeeper配置 -->
<property>
  <name>dfs.ha.automatic-failover.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.ha.automatic-failover.enabled.mycluster</name>
  <value>true</value>
</property>
<!-- 隔离：同一时刻只能有一台服务器对外响应 -->
<property>
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence</value>
</property>
<property>
  <name>dfs.ha.fencing.ssh.private-key-files</name>
  <value>/home/bigdata/.ssh/id_rsa</value>
</property>
<property>
  <name>dfs.datanode.max.xcievers</name>
  <value>4096</value>
</property>

配置$HADOOP_HOME/etc/hadoop/mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
  <name>mapreduce.jobhistory.address</name>
  <value>master:10020</value>
</property>
<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>master:19888</value>
</property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
  <property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>

配置$HADOOP_HOME/etc/hadoop/yarn-site.xml

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>master</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>

复制文件至从节点中

scp -r /home/hadoop/hadoop-3.2.2 hadoop@node1:/home/hadoop/
scp -r /home/hadoop/hadoop-3.2.2 hadoop@node2:/home/hadoop/

执行初始化namenode

存在缓存文件与日志时就删除原数据(所有主机)
目录：/tmp 系统缓存目录下与hadoop相关的文件以及目录
目录：$HADOOP_HOME hadoop安装目录下的logs和tmp两个目录

启动jouranlnode

$HADOOP_HOME/sbin/hadoop-daemons.sh start journalnode

格式化 namenode

$HADOOP_HOME/sbin/hdfs namenode -format

启动 namenode

$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode

同步master主机namenode到从主机上,以下命令在从主机上运行
```
$HADOOP_HOME/sbin/hdfs namenode -bootstrapStandby
```
启动zookeeper集群,以下命令在各个节点以及主节点上执行
```
zkServer.sh start
```
格式化zkfc
```
hdfs zkfc -formatZK
```

停止namenode与journalnode

hadoop-daemon.sh stop namenode
hadoop-daemons.sh stop journalnode