训练二：hadoop安装配置

最新推荐文章于 2022-09-30 10:39:11 发布

xiaotao2018

最新推荐文章于 2022-09-30 10:39:11 发布

阅读量198

点赞数

分类专栏：大数据

大数据专栏收录该内容

10 篇文章

订阅专栏

本文详细介绍了如何配置Hadoop3.1.0集群，包括安装JDK1.8、设置环境变量、配置核心文件如core-site.xml、hdfs-site.xml等，并提供了启动及停止Hadoop集群的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

配置JDK1.8

Hadoop3.1.0要求最低JDK8.0，到官网下载压缩版本解压至任意目录，然后添加环境变量

export JAVA_HOME=/home/hadoop/tools/jdk8
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

以上配置JDK1.8操作需要分别对三台机器同样执行

配置Hadoop3.1.0

配置hadoop环境

# hadoop
export HADOOP_HOME=/home/hadoop/tools/hadoop3
export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

进入目录/hadoop3/etc/hadoop进行一系列配置

**core-site.xml**
<configuration>
  <property>
    <name>fs.defaultFS</name>
     <value>hdfs://master:9000</value>
  </property>
  <property>
      <name>hadoop.tmp.dir</name>
      <value>file:///home/hadoop/tools/hadoop3/tmp</value>
  </property>
</configuration>

**hdfs-site.xml**
<configuration>
<property>
   <name>dfs.replication</name>
   <value>2</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:///home/hadoop/tools/hadoop3/hdfs/name</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:///home/hadoop/tools/hadoop3/hdfs/data</value>
 </property>
 <property>
   <name>dfs.namenode.secondary.http-address</name>
   <value>slave01:9001</value>
 </property>
</configuration>

**yarn-site.xml**
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8025</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8040</value>
    </property>
</configuration>

**mapred-site.xml**
<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

<property>
 <name>mapreduce.application.classpath</name>
 <value>
  /home/hadoop/tools/hadoop3/etc/hadoop,
  /home/hadoop/tools/hadoop3/share/hadoop/common/*,
  /home/hadoop/tools/hadoop3/share/hadoop/common/lib/*,
  /home/hadoop/tools/hadoop3/share/hadoop/hdfs/*,
  /home/hadoop/tools/hadoop3/share/hadoop/hdfs/lib/*,
  /home/hadoop/tools/hadoop3/share/hadoop/mapreduce/*,
  /home/hadoop/tools/hadoop3/share/hadoop/mapreduce/lib/*,
  /home/hadoop/tools/hadoop3/share/hadoop/yarn/*,
  /home/hadoop/tools/hadoop3/share/hadoop/yarn/lib/*
 </value>
</property>
</configuration>

**workers**
注意这边修改的文件就是目录下的workers文件，加入两个slave机器名即可
slave01
slave02

**hadoop-env.sh**
在此文件中加入
export JAVA_HOME=/home/hadoop/tools/jdk8

**yarn-env.sh**
在此文件中加入
export JAVA_HOME=/home/hadoop/tools/jdk8

hadoop启动

/home/hadoop/tools/hadoop3/sbin/start-all.sh
/home/hadoop/tools/hadoop3/sbin/mr-jobhistory-daemon.sh start historyserver

hadoop停止

echo -e "\033[31m ========================================Stopting Hadoop Now !!!======================================= \033[0m"
/home/hadoop/tools/hadoop3/sbin/stop-all.sh
/home/hadoop/tools/hadoop3/sbin/mr-jobhistory-daemon.sh stop historyserver