hadoop 3.2.1集群和2.7 版本部署的区别

1. namenode默认端口号从50070变为9870
2. 群起集群的配置文件从slaves变为workers.
3. hadoop3集群(不是高可用)的配置如下:
   hadoop-env.sh
   mapred-env.sh
   yarn-env.sh
   三个文件添加JAVA_HOME

export JAVA_HOME=/home/andy/work/module/jdk 

core-site.xml:

<configuration>
        <!-- 指定HDFS中NameNode的地址 -->
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://xiaoai04:9000</value>
        </property>

        <!-- 指定Hadoop运行时产生文件的存储目录 -->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/home/andy/work/module/hadoop/data/tmp</value>
        </property>
</configuration>

hdfs-site.xml:

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>

        <!-- 指定Hadoop辅助名称节点主机配置 -->
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>xiaoai06:50090</value>
        </property>
</configuration>

yarn-site.xml:

<configuration>

<!-- Site specific YARN configuration properties -->
<!-- Reducer获取数据的方式 -->
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>xiaoai05</value>
</property>

<!-- 日志聚集功能使能 -->
<property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
</property>
<!-- 日志保留时间设置7天 -->
<property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
</property>
</configuration>

mapred-site.xml:

<configuration>
        <!-- 指定MR运行在Yarn上 -->
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

        <!-- 历史服务器端地址 -->
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>xiaoai04:10020</value>
        </property>

        <!-- 历史服务器web端地址 -->
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>xiaoai04:19888</value>
        </property>

        <!-- 这是3.2以上版本需要增加配置的，不配置运行mapreduce任务可能会有问题，记得使用自己的路径 -->
        <property>
                <name>mapreduce.application.classpath</name>
                <value>
                        /home/andy/work/module/hadoop/etc/hadoop,
                        /home/andy/work/module/hadoop/share/hadoop/common/*,
                        /home/andy/work/module/hadoop/share/hadoop/common/lib/*,
                        /home/andy/work/module/hadoop/share/hadoop/hdfs/*,
                        /home/andy/work/module/hadoop/share/hadoop/hdfs/lib/*,
                        /home/andy/work/module/hadoop/share/hadoop/mapreduce/*,
                        /home/andy/work/module/hadoop/share/hadoop/mapreduce/lib/*,
                        /home/andy/work/module/hadoop/share/hadoop/yarn/*,
                        /home/andy/work/module/hadoop/share/hadoop/yarn/lib/*
                </value>
        </property>
</configuration>

在配置文件夹内创建文件workers,写入一下内容:

xiaoai04
xiaoai05
xiaoai06