Hadoop-HA

最新推荐文章于 2024-08-12 19:00:18 发布

原创最新推荐文章于 2024-08-12 19:00:18 发布 · 186 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #hdfs #zookeeper

该博客详细介绍了如何关闭防火墙并配置Hadoop高可用性(HA)集群，包括修改hdfs-site.xml和core-site.xml文件，设置JournalNodes，启动相关服务，并进行故障切换测试。步骤涵盖从配置文件修改到集群启动的全过程。

关闭防火墙

[root@moyue01 bin]# vim firewalld.sh
#！/bin/bash
for i in moyue01 moyue02 moyue03
do
        ssh $i "systemctl stop firewalld"
done
[root@moyue01 bin]# firewalld.sh

修改hdfs-site.xml

[root@moyue01 hadoop]# pwd
/usr/local/hadoop/etc/hadoop

#添加    
    <property>
        <name>dfs.nameservices</name>
        <value>supercluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.supercluster</name>
        <value>a1,b1</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.supercluster.a1</name>
        <value>moyue01:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.supercluster.b1</name>
        <value>moyue02:8020</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.supercluster.a1</name>
        <value>moyue01:50070</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.supercluster.b1</name>
        <value>moyue02:50070</value>
    </property>
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://moyue01:8485;moyue02:8485;moyue03:8485/journalData</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.supercluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>
                sshfence
                shell(/bin/true)
        </value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_rsa</value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>30000</value>
    </property>
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>

修改core-site.xml

#修改fs.defaultFS
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://supercluster</value>
    </property>
#添加
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/usr/local/hadoop/tmp</value>
    </property>
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>moyue01:2181,moyue02:2181,moyue03:2181</value>
    </property>

配置其他节点

[root@moyue01 hadoop]# scp /usr/local/hadoop/etc/hadoop/* moyue02:/usr/local/hadoop/etc/hadoop/
[root@moyue01 hadoop]# scp /usr/local/hadoop/etc/hadoop/* moyue03:/usr/local/hadoop/etc/hadoop/

启动HA

启动三个节点上的journalnode服务
[root@moyue01 hadoop]# hadoop-daemon.sh start journalnode
[root@moyue02 ~]# hadoop-daemon.sh start journalnode
[root@moyue03 ~]# hadoop-daemon.sh start journalnode
同步日志到journalnode集群上
[root@moyue01 hadoop]# hdfs namenode - initializeSharedEdits
启动以前的节点namenode进程
[root@moyue01 hadoop]# hadoop-daemon.sh start namenode
在新namenode拉取镜像文件
[root@moyue02 hadoop]# hdfs namenode - bootstrapStandby
格式化zkfc
[root@moyue01 hadoop]# zkServer.sh start
[root@moyue02 hadoop]# zkServer.sh start
[root@moyue03 hadoop]# zkServer.sh start
[root@moyue01 hadoop]# hdfs zkfc -formatZK

测试

测试
start-dfs.sh
[root@moyue01 hadoop]# jps
12936 QuorumPeerMain
26360 Jps
24393 NameNode
25195 DFSZKFailoverController
24893 JournalNode
24575 DataNode
[root@moyue02 logs]# jps
22384 JournalNode
22004 NameNode
22183 DataNode
22604 DFSZKFailoverController
11391 QuorumPeerMain
23615 Jps
Overview 'moyue01:8020' (active)
Overview 'moyue02:8020' (standby)

[root@moyue01 hadoop]# jps
12936 QuorumPeerMain
26360 Jps
24393 NameNode
25195 DFSZKFailoverController
24893 JournalNode
24575 DataNode
[root@moyue01 hadoop]# kill -9 24393
Overview 'moyue02:8020' (active)