Hadoop core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml在伪分布模型下、完全分布模式下配置

一、基于伪分布模式下(基于完全分布模式下)

(一)、core-site.xml

Hadoop的core-site.xml配置

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop1:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/export/data/hadoop-wfb-3.3.0</value> (基于伪分布模式下)
        <value>/export/data/hadoop-3.3.0</value> (基于完全分布模式下)
</property>
<property>
        <name>hadoop.http.staticuser.user</name>
        <value>root</value>
</proxyuser>
<property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>*</value>
</property>
<property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
</property>
<property>
        <name>fs.trash.interval</name>
        <value>1440</value>
</property>

各配置的作用

1. fs.defaultFS 

2. hadoop.tmp.dir

"wfb代表用户名"，应该将路径中的"wfb"改为你自己的用户名。

3. hadoop.http.staticuser.user 

4.hadoop.proxyuser.root.hosts

5.hadoop.proxyuser.root.groups

(二)、hdfs-site.xml

Hadoop的hdfs-site.xml配置

<property>
        <name>dfs.replication</name>
        <value>1</value> (基于伪分布模式下)

        <value>2</value> (基于完全分布模式下)
</property>
<property>
        <name>dfs.namenode.secondary.http-address</name> 
        <value>hadoop1:9868</value> (基于伪分布模式下)
        <value>hadoop2:9868</value> (基于完全分布模式下)
</property>

各配置的作用 

1.dfs.replication

2.dfs.namenode.secondary.http-address 

(三)、mapred-site.xml

Hadoop的mapred-site.xml配置(Hadoop基于完全分布模式下)

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoopl:10020</value>
<property>
<property>    
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoopl:19888</value>
</property>
<property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=/export/server/wfb-hadoop/hadoop-3.3.0</value> (基于伪分布模式下)

        <value>HADOOP_MAPRED_HOME=${HADOOP_HOE}</value> (基于完全分布模式下)
</property>
<property>
        <name>mapreduce.map.env</name>

         <value>HADOOP_MAPRED_HOME=/export/server/wfb-hadoop/hadoop-3.3.0</value> (基于伪分布模式下)
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> (基于完全分布模式下)        
</property>
<property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=/export/server/wfb-hadoop/hadoop-3.3.0</value> (基于伪分布模式下)

        <value>HADOOP MAPRED HOME=${HADOOP HOME }</value> (基于完全分布模式下)        
</property>

各配置的作用 

1.mapreduce.framework.name

2.mapreduce.jobhistory.address

3.mapreduce.jobhistory.webapp.address

4.yarn.app.mapreduce.am.env

5.mapreduce.map.env

6.mapreduce.reduce.env

(四)、yarn-site.xml

Hadoop的yarn-site.xml的配置(基于伪分布模式下与基于完全分布模式下相同)

<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoopl</value>
</property>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
</property>
<property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
</property>
<property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
</property>
<property>
        <name>yarn.log.server.url</name>
        <value>http://hadoopl:19888/jobhistory/logs</value></property>
<property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
</property>

各配置的作用 

1.yarn.resourcemanager.hostname

2.yarn.nodemanager.aux-services

3.yarn.nodemanager.pmem-check-enabled

4.yarn.nodemanager.vmem-check-enabled

5.yarn.log-aggregation-enable

6.yarn.log.server.url

7.yarn.log-aggregation.retain-seconds