CentOS6.5环境部署Hadoop2.8.1集群（HA）

部署前提
1、三台主机安装JDK1.7
2、关闭防火墙、selinux
3、配置静态IP

一、所用软件
hadoop-2.8.1
zookeeper-3.4.6

hadoop-2.8.1下载地址
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.8.1/
zookeeper-3.4.6 下载地址
http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.6/

二、机器规划(对应关系写入到/etc/hosts)

IP	HOST	Install Soft	Daemon
192.168.95.10	hadoop000	Hadoop、zookeeper	NameNode、 DFSZKFailoverController、 JournalNode、 DataNode 、ResourceManager 、JobHistoryServer、 NodeManager、 QuorumPeerMain
192.168.95.20	hadoop001	Hadoop、zookeeper	NameNode、DFSZKFailoverController、JournalNode、DataNode、ResourceManager、NodeManager、QuorumPeerMain
192.168.95.30	hadoop002	Hadoop、zookeeper	JournalNode、DataNode、NodeManager、QuorumPeerMain

三、创建目录并上传Hadoop、zookeeper
/opt/目录下创建soft文件夹(三台)

mkdir   /opt/soft

上传hadoop-2.8.1、zookeeper-3.4.6 至hadoop000机器/opt/soft目录，解压

 tar -zxvf zookeeper-3.4.6.tar.gz
 tar -zxvf hadoop-2.8.1.tar.gz

重命名

 mv hadoop-2.8.1 hadoop
 mv zookeeper-3.4.6 zookeeper

查看

[root@hadoop000 soft]# ls
hadoop  hadoop-2.8.1.tar.gz  zookeeper  zookeeper-3.4.6.tar.gz

四、三台主机SSH互相信任关系配置

请参照http://blog.youkuaiyun.com/gavin_chun/article/details/77924405来配置

五、部署zookeeper
1、设置zookeeper环境变量

vim /etc/profile
export ZOOKEEPER_HOME=/opt/soft/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH
生效环境变量
source /etc/profile
验证
[root@hadoop000 zookeeper]# echo $ZOOKEEPER_HOME
/opt/soft/zookeeper

2、进入/opt/soft/zookeeper/conf目录
将目录下默认文件zoo_sample.cfg拷贝一份并命令成zoo.cfg

[root@hadoop000 conf]# cp zoo_sample.cfg zoo.cfg

3、编辑 zoo.cfg文件添加以下内容

dataDir=/opt/soft/zookeeper/data
server.1=hadoop000:2888:3888
server.2=hadoop001:2888:3888
server.3=hadoop002:2888:3888

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.

dataDir=/opt/soft/zookeeper/data

# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1


server.1=hadoop000:2888:3888
server.2=hadoop001:2888:3888
server.3=hadoop002:2888:3888

4、进入zookeeper目录

[root@hadoop000 zookeeper]# mkdir data
[root@hadoop000 zookeeper]# touch data/myid 
[root@hadoop000 zookeeper]# echo 1 > data/myid 

5、将hadoop000的zookeeper传到hadoop001/002/opt/soft/目录

[root@hadoop000 soft]# scp -r zookeeper 192.168.95.20:/opt/soft/ 
[root@hadoop000 soft]# scp -r zookeeper 192.168.95.30:/opt/soft/  

hadoop001/002做如下操作
[root@hadoop001 zookeeper]# echo 2 > data/myid
[root@hadoop002 zookeeper]# echo 3 > data/myid 

六、部署hadoop
1、部署hadoop环境变量，查看hadoop目录

[root@hadoop000 hadoop]# pwd
/opt/soft/hadoop

编辑/etc/profile,添加以下内容

export HADOOP_HOME=/opt/soft/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

验证

[root@hadoop000 hadoop]# echo $HADOOP_HOME
/opt/soft/hadoop

2、编辑hadoop-env.sh
进入/opt/soft/hadoop/etc/hadoop目录，在hadoop-env.sh添加以下内容（25行开始）

25 export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
26 export  HADOOP_OPTS="$HADOOP_OPTS  -Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/bin/native"

3、修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml四个文件

具体修改内容值请参考http://blog.youkuaiyun.com/gavin_chun/article/details/77936475

将/opt/soft/hadoop/etc/hadoop目录下原有四个文件删除，将配置好的四个文件放到这个目录即可
注意：文件有些配置内容以自己机器配置为准！

4、创建临时文件夹和分发文件夹

[root@hadoop000 hadoop]# mkdir -p /opt/soft/hadoop/tmp
[root@hadoop000 hadoop]# chmod -R 777 /opt/soft/hadoop/tmp
[root@hadoop000 hadoop]# chown -R root:root /opt/soft/hadoop/tmp

[root@hadoop000 hadoop]# scp -r hadoop root@hadoop001:/opt/soft
[root@hadoop000 hadoop]# scp -r hadoop root@hadoop002:/opt/soft

七、启动集群
1、启动zookeeper

[root@hadoop000 zookeeper]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@hadoop001 zookeeper]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@hadoop002 zookeeper]# $ZOOKEEPER_HOME/bin/zkServer.sh start

2、启动journalnode进程

[root@hadoop000 ~]# cd /opt/soft/hadoop/sbin
[root@hadoop000 sbin]# hadoop-daemon.sh start journalnode

[root@hadoop001 ~]# cd /opt/soft/hadoop/sbin
[root@hadoop001 sbin]# hadoop-daemon.sh start journalnode

[root@hadoop002 ~]# cd /opt/soft/hadoop/sbin
[root@hadoop002 sbin]# hadoop-daemon.sh start journalnode

3、Namenode格式化
[root@hadoop000 hadoop]# hadoop namenode -format
……………..
……………..
17/09/02 23:16:50 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1577237506-192.168.137.130-1504365410166
17/09/02 23:16:50 INFO common.Storage: Storage directory /opt/soft/hadoop/data/dfs/name has been successfully formatted.
17/09/02 23:16:50 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/soft/hadoop/data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/09/02 23:16:50 INFO namenode.FSImageFormatProtobuf: Image file /opt/soft/hadoop/data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 306 bytes saved in 0 seconds.
17/09/02 23:16:51 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/09/02 23:16:51 INFO util.ExitUtil: Exiting with status 0
17/09/02 23:16:51 INFO namenode.NameNode: SHUTDOWN_MSG:
/**************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop001/192.168.95.10

---

同步hadoop000元数据到hadoop001(Namenode standby)

[root@hadoop000 hadoop]# scp -r data/ root@hadoop001:/opt/soft/hadoop

4、zookeeper初始化
[root@hadoop000 bin]# hdfs zkfc -formatZK
……………..
……………..
17/09/02 23:19:13 INFO ha.ActiveStandbyElector: Session connected.
17/09/02 23:19:13 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK.
17/09/02 23:19:13 INFO zookeeper.ZooKeeper: Session: 0x35e42f121f50000 closed
17/09/02 23:19:13 INFO zookeeper.ClientCnxn: EventThread shut down
17/09/02 23:19:13 INFO tools.DFSZKFailoverController: SHUTDOWN_MSG:
/**************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at hadoop001/192.168.95.10
**************************************************/

5、启动HDFS

[root@hadoop000 sbin]# start-dfs.sh
Starting namenodes on [hadoop000 hadoop001]
hadoop000: starting namenode, logging to /opt/soft/hadoop/logs/hadoop-root-namenode-hadoop000.out
hadoop001: starting namenode, logging to /opt/soft/hadoop/logs/hadoop-root-namenode-hadoop001.out
hadoop001: starting datanode, logging to /opt/soft/hadoop/logs/hadoop-root-datanode-hadoop001.out
hadoop000: starting datanode, logging to /opt/soft/hadoop/logs/hadoop-root-datanode-hadoop000.out
hadoop002: starting datanode, logging to /opt/soft/hadoop/logs/hadoop-root-datanode-hadoop002.out
Starting journal nodes [hadoop000 hadoop001 hadoop002]
hadoop000: starting journalnode, logging to /opt/soft/hadoop/logs/hadoop-root-journalnode-hadoop000.out
hadoop001: starting journalnode, logging to /opt/soft/hadoop/logs/hadoop-root-journalnode-hadoop001.out
hadoop002: starting journalnode, logging to /opt/soft/hadoop/logs/hadoop-root-journalnode-hadoop002.out
Starting ZK Failover Controllers on NN hosts [hadoop000 hadoop001]
hadoop001: starting zkfc, logging to /opt/soft/hadoop/logs/hadoop-root-zkfc-hadoop001.out
hadoop000: starting zkfc, logging to /opt/soft/hadoop/logs/hadoop-root-zkfc-hadoop000.out

6、启动YARN

[root@hadoop000 sbin]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/software/hadoop/logs/yarn-root-resourcemanager-hadoop000.out
hadoop002: starting nodemanager, logging to /opt/software/hadoop/logs/yarn-root-nodemanager-hadoop002.out
hadoop001: starting nodemanager, logging to /opt/software/hadoop/logs/yarn-root-nodemanager-hadoop001.out
hadoop000: starting nodemanager, logging to /opt/software/hadoop/logs/yarn-root-nodemanager-hadoop000.out

7、备机启动Resourcemanager

[root@hadoop001 hadoop]# yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /opt/soft/hadoop/logs/yarn-root-resourcemanager-hadoop002.out

8、查看进程

[root@hadoop000 ~]# jps
2488 JournalNode
2641 DFSZKFailoverController
2808 ResourceManager
2280 DataNode
5395 Jps
2179 NameNode
2910 NodeManager
2030 QuorumPeerMain

[root@hadoop001 ~]# jps
1927 QuorumPeerMain
2583 ResourceManager
2403 NodeManager
6255 Jps
2262 DFSZKFailoverController
2173 JournalNode
1998 NameNode
2063 DataNode

[root@hadoop002 ~]# jps
2049 NodeManager
2678 Jps
1921 JournalNode
1851 DataNode
1786 QuorumPeerMain

八、验证HDFS/YARN的HA
hadoop000 (Namenode active)

http://192.168.95.10:50070/dfshealth.html#tab-overview

hadoop001 （Namenode standby）

http://192.168.95.10:50070/dfshealth.html#tab-overview


hadoop000 （ResourceManager active）
http://192.168.95.10:8088/cluster/cluster

hadoop001 （ResourceManager standby）
http://192.168.95.20:8088/cluster/cluster