大数据完全式搭建_gateway=dns1= dns2=-优快云博客

本文链接：https://blog.youkuaiyun.com/weixin_63168851/article/details/129958524

本文描述了在Linux环境中配置和安装ZooKeeper以及创建Hadoop高可用(HA)集群的过程。包括修改网络配置、关闭防火墙与Selinux、卸载并安装JDK、配置环境变量、设置主机名与IP、实现SSH免密码登录、部署ZooKeeper、配置ZooKeeper集群、解压与配置Hadoop、修改Hadoop配置文件、启动服务等步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、修改网络与主机名
vim /etc/hostname
[root@syh ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
       OTPROTO="static"
       ONBOOT="yes"
       IPADDR=192.168.74.17
       PREFIX=24
       GATEWAY=192.168.74.1
       DNS1=223.6.6.6
       DNS2=119.29.29.29

2、刷新网络
[root@syh ~]# systemctl restart network

3、关闭防火墙与selinux
[root@syh ~]# systemctl stop|disabled firewalld.service
[root@syh ~]# setenforce 0 临时关闭
[root@syh ~]# vim /etc/selinux/config 修改：SELINUX=disabled 永久关闭

3.1、删除系统的jdk
[root@syh ~]# rpm -qa |grep openjdk 删除
rpm -e --nodeps ()
3.2、解压jdk
[root@syh packages]# tar -zxvf jdk-8u211-linux-x64.tar.gz -C /opt/programs

3.3、修改/etc/profile ,配置jdk系统环境变量
   vim /etc/profile
       加入代码：export JAVA_HOME=/opt/programs/jdk1.8.0_211
                  export PATH=$PATH:$JAVA_HOME/bin

3.4、刷新profile文件
source /etc/profile

3.5、执行 java -version

4、克隆2台，修改主机名，ip
主机名 vim /etc/hostname
ip vim /etc/sysconfig/network-scripts/ifcfg-ens33

5、配置3个节点主机ip映射
vim /etc/hosts
ip 主机名

6、3台电脑，SSH免密码登录
ssh-keygen(过程中遇到询问三个回车）
ssh-copy-id 主机名1
ssh-copy-id 主机名2
ssh-copy-id 主机名3

7、zookeeper放到vm虚拟机中，winSCP传输过去 cd /opt/packages

8、解压tar -zxvf zookeeper-3.4.12.tar.gz -C /opt/programs

9、改变路径到zookeeper：cd /opt/programs/zookeeper-3.4.12

10、创建文件夹：mkdir data
mkdir logs

11、改变路径到data：cd data
   1写入到myid文件：echo '1'>myid 或者 vi /etc/myid

12、改变路径到conf文件： cd /opt/programs/zookeeper-3.4.12/conf
   重命名：cp zoo_sample.cfg zoo.cfg

13、修改zoo.cfg：vi zoo.cfg
   修改：dataDir=/opt/programs/zookeeper-3.4.12/data
   增加：dataLogDir=/opt/programs/zookeeper-3.4.12/logs
          server.1=syh:2888:3888
          server.2=syh1:2888:3888
          server.3=syh2:2888:3888

14、ZooKeeper安装目录远程复制到syh1和syh2： (我的问题，复制过去的代码data有错误)
scp -r /opt/programs/zookeeper-3.4.12 root@syh1:/opt/programs/
scp -r /opt/programs/zookeeper-3.4.12 root@syh2:/opt/programs/

15、修改syh1和syh2的myid编号分别为2和3：
vim /opt/programs/zookeeper-3.4.12/data/myid

16、在三个节点分别进行相同操作：
   vim /etc/profile
       export ZOOKEEPER_HOME=/opt/programs/zookeeper-3.4.12
       export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin
   刷新：source /etc/profile

17、在三个节点分别进行相同操作：启动ZooKeeper
   zkServer.sh start

18、在三个节点分别进行相同操作：查看ZooKeeper
   zkServer.sh status

19、hadoop包上传到packages，进入该目录
   cd /opt/packages

20、解压到/opt/programs
   tar -zxvf hadoop-2.7.7.tar.gz -C /opt/programs

21、依次修改core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml slaves hadoop-env.sh mapred-env.sh yarn-env.sh
   进入该目录：cd /opt/programs/hadoop-2.7.7/etc/hadoop/
core-site.xml:
       <configuration>
           <property>
               <name>fs.defaultFS</name>
               <value>hdfs://ns</value>
           </property>
           <property>
               <name>hadoop.tmp.dir</name>
               <value>/opt/programs/hadoop-2.7.7/tmp</value>
           </property>
               <property>
                       <name>ha.zookeeper.quorum</name>
                       <value>syh1:2181,syh2:2181,syh3:2181</value>
           </property>
       </configuration>

hdfs-site.xml
       <configuration>
               <property>
                       <name>dfs.replication</name>
                       <value>3</value>
               </property>
               <property>
                       <name>dfs.nameservices</name>
                       <value>ns</value>
               </property>
               <property>
                       <name>dfs.ha.namenodes.ns</name>
                       <value>nn1,nn2</value>
               </property>
               <property>
                       <name>dfs.namenode.rpc-address.ns.nn1</name>
                       <value>syh1:9000</value>
               </property>
               <property>
                       <name>dfs.namenode.http-address.ns.nn1</name>
                       <value>syh1:50070</value>
               </property>
           <property>
                       <name>dfs.namenode.rpc-address.ns.nn2</name>
                       <value>syh2:9000</value>
               </property>
           <property>
                       <name>dfs.namenode.http-address.ns.nn2</name>
                       <value>syh2:50070</value>
               </property>
           <property>
                       <name>dfs.namenode.shared.edits.dir</name>
                       <value>qjournal://syh1:8485;syh2:8485;syh3:8485/ns</value>
               </property>
           <property>
                       <name>dfs.journalnode.edits.dir</name>
                       <value>/opt/programs/hadoop-2.7.7/journal/data</value>
               </property>
           <property>
                       <name>dfs.ha.automatic-failover.enabled</name>
                       <value>true</value>
               </property>
           <property>
                       <name>dfs.client.failover.proxy.provider.ns</name>
                       <value>org.apache.hadoop.hafs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
               </property>
           <property>
                       <name>dfs.ha.fencing.methods</name>
                       <value>
                   sshfence
                   shell(/bin/true)
               </value>
               </property>
           <property>
                       <name>dfs.ha.fencing.ssh.private-key-files</name>
                       <value>/root/.ssh/id_rsa</value>
               </property>
           <property>
                       <name>dfs.ha.fencing.ssh.connect-timeout</name>
                       <value>30000</value>
               </property>
       </configuration>

mv mapred-site.xml.template mapred-site.xml
       <configuration>
           <property>
                       <name>mapreduce.framework.name</name>
                       <value>yarn</value>
               </property>
       </configuration>

yarn-site.xml
       <configuration>
       
               <property>
                       <name>yarn.resourcemanager.ha.enabled</name>
                       <value>true</value>
               </property>
               <property>
                       <name>yarn.resourcemanager.cluster-id</name>
                       <value>yrc</value>
               </property>
               <property>
                       <name>yarn.resourcemanager.ha.rm-ids</name>
                       <value>rm1,rm2</value>
               </property>
               <property>
                       <name>yarn.resourcemanager.hostname.rm1</name>
                       <value>syh1</value>
               </property>
               <property>
                       <name>yarn.resourcemanager.hostname.rm2</name>
                       <value>syh2</value>
               </property>
               <property>
                       <name>yarn.resourcemanager.zk-address</name>
                       <value>syh1:2181,syh2:2181,syh3:2181</value>
               </property>
               <property>
                       <name>yarn.nodemanager.aux-services</name>
                       <value>mapreduce_shuffle</value>
               </property>
       </configuration>

vim slaves
       syh1
       syh2
       syh3

在hadoop-env.sh , mapred-env.sh , yarn-env.sh 均加入
       export JAVA_HOME=/opt/programs/jdk1.8.0_211

22、将syh1节点远程复制到syh2和syh3
scp -r /opt/programs/hadoop-2.7.7 root@syh2:/opt/programs/
scp -r /opt/programs/hadoop-2.7.7 root@syh3:/opt/programs/

23、在三台节点修改/etc/profile
   vim /etc/profile
       export HADOOP_HOME=/opt/programs/hadoop-2.7.7
       export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   source /etc/profile

24、启动测试
   hadoop-daemon.sh start journalnode (3个节点分别启动)
   hdfs namenode -format

25、进入hadoop,复制tmp到syh2节点
   cd /opt/programs/hadoop-2.7.7
   scp -r tmp/ root@syh2:/opt/programs/hadoop-2.7.7

26、syh1节点格式化ZKFC
   hdfs zkfc -formatZK

27、syh1节点格式化
   start-dfs.sh
   start-yarn.sh
28、jps