hadoop笔记之部署hadoop集群(HA)+yarn

部署hadoop集群(HA)+yarn

前提条件

部署虚拟机：有以下条件

• 配置网络
• 设置USEDNS为no
• 关闭防火墙
• 关闭SELinux
• 配置免秘钥登录
• 安装jdk

服务器	nn	dn	jnn	rm	nn	zk	zkfc
node1	✔		✔				✔
node1	✔	✔	✔		✔	✔	✔
node3		✔	✔	✔	✔	✔
node4		✔		✔	✔	✔

zookeeper集群搭建

安装到node2,node3,node4

a) 上传安装包

将zookeeper.tar.gz上传到node2、node3、node4

b) 解压

tar -zxf zookeeper-3.4.6.tar.gz -C /opt

c) 配置环境变量

在/etc/profile最后添加:

vim /etc/profile

export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.6
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin

复制到其他机器上,然后让配置生效

for node in node2 node3 node4;do scp /etc/profile $node:/etc/;done
#所有机器上执行
. /etc/profile

d) zoo.cfg

到$ZOOKEEPER_PREFIX/conf下

复制zoo_sample.cfg为zoo.cfg

#cd /opt/zookeeper-3.4.6/conf/
cp /opt/zookeeper-3.4.6/conf/zoo_sample.cfg  /opt/zookeeper-3.4.6/conf/zoo.cfg
vim /opt/zookeeper-3.4.6/conf/zoo.cfg

编辑zoo.cfg
添加如下行：

server.1=node2:2881:3881
server.2=node3:2881:3881
server.3=node4:2881:3881

修改

dataDir=/var/zookeeper/data

将/opt/zookeeper-3.4.6通过网络拷贝到node2、node3上

for node in node2 node3 node4;do scp -r /opt/zookeeper-3.4.6/conf/zoo.cfg $node:/opt/zookeeper-3.4.6/conf;done

e) 创建myid

创建/var/zookeeper/data目录，并在该目录下放一个文件：myid
在myid中写下当前zookeeper的编号

#node2
mkdir -p /var/zookeeper/data
echo 1 > /var/zookeeper/data/myid
#node3
mkdir -p /var/zookeeper/data
echo 2 > /var/zookeeper/data/myid
#node4
mkdir -p /var/zookeeper/data
echo 3 > /var/zookeeper/data/myid

f) 测试

启动命令:

zkServer.sh start|stop|status

查看日志

less zookeeper.out

安装hadoop

1.上传hadoop安装包

2.在所有机器上解压

tar -zxf hadoop-2.6.5.tar.gz -C /opt

3.配置全局环境变量

vi /etc/profile

​ 添加两行记录：

export HADOOP_PREFIX=/opt/hadoop-2.6.5
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin

​ 复制到其他机器上

for node in node2 node3 node4;do scp /etc/profile $node:/etc/;done

在所有机器上执行执行. /etc/profile让配置生效

source /etc/profile

修改配置文件

1.配置hadoop的java_home

修改/opt/hadoop-2.6.5/etc/hadoop/hadoop-env.sh

vi /opt/hadoop-2.6.5/etc/hadoop/hadoop-env.sh

添加

export JAVA_HOME=/usr/java/jdk1.8.0_172-amd64

2.NameNode的slaves

vi /opt/hadoop-2.6.5/etc/hadoop/slaves

添加

node1
node2
node3
node4

3.core-site.xml

vim cd /opt/hadoop-2.6.5/etc/hadoop/core-site.xml

##########添加##############
<configuration>
  <property>
    <