hadoop高可用

最新推荐文章于 2025-05-12 14:54:47 发布

原创最新推荐文章于 2025-05-12 14:54:47 发布 · 456 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #高可用

oo 专栏收录该内容

28 篇文章

订阅专栏

本文介绍了在分布式Hadoop基础上实现高可用性的步骤，包括Zookeeper的安装和配置，以及详细列出的Hadoop集群服务器节点信息。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

在分布式hadoop的基础上：
安装zookeeper：

[hadoop@server1 ~]$ tar zookeeper-3.4.9.tar.gz
[hadoop@server1 ~]$ ls
hadoop               java                       zookeeper-3.4.9
hadoop-2.7.3.tar.gz  jdk-7u79-linux-x64.tar.gz  zookeeper-3.4.9.tar.gz
[hadoop@server1 ~]$ cd zookeeper-3.4.9
[hadoop@server1 zookeeper-3.4.9]$ cd conf/
[hadoop@server1 conf]$ mv zoo_sample.cfg zoo.cfg
[hadoop@server1 conf]$ ls
configuration.xsl  log4j.properties  zoo.cfg
[hadoop@server1 conf]$ vim zoo.cfg 

------------------
server.1=172.25.5.2:2888:3888
server.2=172.25.5.3:2888:3888
server.3=172.25.5.4:2888:3888

server2:

[hadoop@server2 hadoop]$ cd /tmp/zookeeper/    建立
[hadoop@server2 zookeeper]$ cat myid 
1
[hadoop@server2 zookeeper]$

server2:

[hadoop@server3 hadoop]$ cd /tmp/zookeeper/    建立
[hadoop@server3 zookeeper]$ cat myid 
2
[hadoop@server3 zookeeper]$

server4:

[hadoop@server4 hadoop]$ cd /tmp/zookeeper/    建立
[hadoop@server4 zookeeper]$ cat myid 
3
[hadoop@server4 zookeeper]$

server2,3,4:

[hadoop@server2 zookeeper-3.4.9]$ bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[hadoop@server2 zookeeper-3.4.9]$ bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: follower
[hadoop@server2 hadoop]$ sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-journalnode-server2.out
[hadoop@server2 hadoop]$ jps
3116 QuorumPeerMain
3625 Jps
3445 JournalNode

hadoop配置：

[hadoop@server1 hadoop]$ vim etc/hadoop/core-site.xml 
-------------------
<configuration>
<property>
        <name>fs.defaultFS</name>
                        <value>hdfs://masters</value>
                                            </property>

<property>
<name>ha.zookeeper.quorum</name>
<value>172.25.5.2:2181,172.25.5.3:2181,172.25.5.4:2181</value>
</property>

</configuration>

[hadoop@server1 hadoop]$ vim etc/hadoop/hdfs-site.xml
----------------------
<configuration>
<property>
        <name>dfs.replication</name>
                        <value>3</value>
                                            </property>

<property>
<name>dfs.nameservices</name>
<value>masters</value>
</property>

<property>
<name>dfs.ha.namenodes.masters</name>
<value>h1,h2</value>
</property>

<property>
<name>dfs.namenode.rpc-address.masters.h1</name>
<value>172.25.5.1:9000</value>
</property>

<property>
<name>dfs.namenode.http-address.masters.h1</name>
<value>172.25.5.1:50070</value>
</property>

<property>
<name>dfs.namenode.rpc-address.masters.h2</name>
<value>172.25.5.5:9000</value>
</property>

<property>
<name>dfs.namenode.http-address.masters.h2</name>
<value>172.25.5.5:50070</value>
</property>

<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://172.25.5.2:8485;172.25.5.3:8485;172.25.5.4:8485/masters</value>
</property>

<property>
<name>dfs.journalnode.edits.dir</name>
<value>/tmp/journaldata</value>
</property>

<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

<property>
<name>dfs.client.failover.proxy.provider.masters</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>

<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>

<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>

[hadoop@server1 hadoop]$ vim etc/hadoop/slaves
------------------
172.25.5.2
172.25.5.3
172.25.5.4

[hadoop@server1 hadoop]$ bin/hdfs namenode -format
[hadoop@server1 hadoop]$ bin/hdfs zkfc -formatZK

server1：172.25.5.1

[hadoop@server1 hadoop]$ sbin/start-dfs.sh
Starting namenodes on [server1 server5]
server1: starting namenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-namenode-server1.out
server5: starting namenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-namenode-server5.out
172.25.5.2: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-server2.out
172.25.5.4: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-server4.out
172.25.5.3: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-server3.out
Starting journal nodes [172.25.5.2 172.25.5.3 172.25.5.4]
172.25.5.2: starting journalnode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-journalnode-server2.out
172.25.5.3: starting journalnode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-journalnode-server3.out
172.25.5.4: starting journalnode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-journalnode-server4.out
Starting ZK Failover Controllers on NN hosts [server1 server5]
server1: starting zkfc, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-zkfc-server1.out
server5: starting zkfc, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-zkfc-server5.out
[hadoop@server1 hadoop]$ jps
7122 NameNode
7491 Jps
7400 DFSZKFailoverController
[hadoop@server1 hadoop]$

server5：172.25.5.5

[hadoop@server5 ~]$ jps
2338 DFSZKFailoverController
2411 Jps
2259 NameNode
[hadoop@server5 ~]$

server2：172.25.5.2

[hadoop@server2 hadoop]$ jps
3149 DataNode
3242 JournalNode
2074 QuorumPeerMain
3305 Jps
[hadoop@server2 hadoop]$

server3：172.25.5.3

[hadoop@server3 hadoop]$ jps
3048 Jps
2985 JournalNode
2892 DataNode
1842 QuorumPeerMain
[hadoop@server3 hadoop]$

server4：172.25.5.4

[hadoop@server4 hadoop]$ jps
2989 JournalNode
3051 Jps
1701 QuorumPeerMain
2896 DataNode
[hadoop@server4 hadoop]$

这里写图片描述

[hadoop@server1 hadoop]$ bin/hdfs dfs -ls
ls: `.': No such file or directory
[hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir /user
[hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop
[hadoop@server1 hadoop]$ bin/hdfs dfs -ls
[hadoop@server1 hadoop]$ bin/hdfs dfs -put input/
[hadoop@server1 hadoop]$ bin/hdfs dfs -ls
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2018-08-28 09:41 input
[hadoop@server1 hadoop]$ jps
7639 NameNode
8165 Jps
7400 DFSZKFailoverController
[hadoop@server1 hadoop]$ kill -9 7639
[hadoop@server1 hadoop]$ jps
8196 Jps
7400 DFSZKFailoverController
[hadoop@server1 hadoop]$ bin/hdfs dfs -ls
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2018-08-28 09:41 input
[hadoop@server1 hadoop]$ 
[hadoop@server1 hadoop]$

这里写图片描述

[hadoop@server1 hadoop]$ sbin/stop-dfs.sh
Stopping namenodes on [server1 server5]
server1: stopping namenode
server5: stopping namenode
172.25.5.2: stopping datanode
172.25.5.3: stopping datanode
172.25.5.4: stopping datanode
Stopping journal nodes [172.25.5.2 172.25.5.3 172.25.5.4]
172.25.5.3: stopping journalnode
172.25.5.4: stopping journalnode
172.25.5.2: stopping journalnode
Stopping ZK Failover Controllers on NN hosts [server1 server5]
server5: stopping zkfc
server1: stopping zkfc
[hadoop@server1 hadoop]$