搭建全分布式HADOOP集群_轻薄本能跑集群吗-优快云博客

本文链接：https://blog.youkuaiyun.com/YongYu_IT/article/details/112743918

1、安装jdk
（1）解压缩jdk包

tar xzvf jdk-8u211-linux-x64.tar.gz

（2）配置JAVA环境变量并验证jdk是否安装成功

vi ~/.bashrc (~ 表示当前用户目录如 /home/yong)

export JAVA_HOME=/home/yong/jdk-8u211-linux-x64/jdk1.8.0_211
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin

source ~/.bashrc

java -version
javac -version

2、安装hadoop并验证hadoop是否安装成功

tar xzvf hadoop-3.1.2.tar.gz

vi ~/.bashrc
export HADOOP_HOME=/home/yong/hadoop-3.1.2
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

hadoop version

hdfs

hadoop

3、修改每个节点的IP（每个IP取别名，所有机器都要处理，ip和别名直接间隔type键）

修改 /etc/hostname 文件

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

4、SSH免密登录

ssh-keygen -t rsa -N "" --生产公钥
ssh-copy-id -i ~/.ssh/id_rsa.pub yong@master
ssh-copy-id -i ~/.ssh/id_rsa.pub yong@node1
ssh-copy-id -i ~/.ssh/id_rsa.pub yong@node2

ping master

ping node1

ping node2

5、配置master节点

cd $HADOOP_HOME
mkdir my_config
vi ~/.bashrc
export HADOOP_CONF_DIR=$HADOOP_HOME/my_config
export HADOOP_LOG_DIR=$HADOOP_HOME/my_log
source ~/.bashrc
cd $HADOOP_CONF_DIR
cp -r $HADOOP_HOME/etc/hadoop/* .

（1）修改core-site.xml文件

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://mymaster/</value>
  </property>

（2）修改 hdfs-site.xml文件

  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>

（3）修改mapred-site.xml文件

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

（4）修改yarn-site.xml文件

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

6、配置worker节点