Hadoop配置文件 HDFS垃圾回收，无密码登录和各个服务组件如何配置那台服务器运行并测试

默认配置文件

在share/hadoop 里面的jar包里面

core-default.xml
hdfs-default.xml
yarn-default.xml
mapred-default.xml

自定义配置文件$HADOOP_HOME/etc/hadoop/

core-site.xml
hdfs-site.xml
yarn-site.xml
mapred-site.xml

core-site.xml

 <property>
                   <name>fs.trash.interval</name>
        <value>7*24*60</value>
</property>

===========================启动方式=================

启动方式：都在sbin目录下

各个服务组件逐一启动

1.hdfs(hadoop-daemon.sh start|stop namenode|datanode|secondarynamenode)

2.yarn(yarn-daemon.sh start|stop resourcemanager|nodemanager)

3.mapreduce(mr-historyserver-daemon.sh start|stop historyserver)

各个模块分开启动(需要配置，无密码登录)

无密码登录设置
1.根目录 ls -a
2.cd .ssh/
3.ssh-keygen -t rsa 生成公私钥
4.ssh-copy-id hadoop-senior.cj.com 拷贝到远程
5.输yes, 输入远程密码
6.测试无密码登录 ssh hadoop-senior.cj.com

hdfs（start-dfs.sh stop-dfs.sh）
yarn (start-yarn.sh stop-yarn.sh)

全部启动(sbin)不推荐

start-all.sh
stop-all.sh

===================================

相关配置的意义

HDFS

NameNode是哪个属性指定的

 <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop-senior.cj.com:8020</value>
    </property>

DataNodes

slaves文件里面 hadoop-senior.cj.com

secondaryNode

是由hdfs-site.xml里面配置的

 <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop-senior.cj.com:50090</value>
    </property>

配置好了要关闭
sbin/stop-dfs.sh,就可以关掉namenode,datanode,secondarynode,如果是分布式集群还可以看到是哪个服务器在关闭

[cj@hadoop-senior hadoop-2.5.0]$ sbin/stop-dfs.sh
17/11/10 08:14:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [hadoop-senior.cj.com]
hadoop-senior.cj.com: stopping namenode
hadoop-senior.cj.com: stopping datanode
Stopping secondary namenodes [hadoop-senior.cj.com]
hadoop-senior.cj.com: no secondarynamenode to stop
17/11/10 08:14:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

YARN

ResourceManager

yarn-site.xml

<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-senior.cj.com</value>
    </property>

DataNodes

slaves文件里面 hadoop-senior.cj.com

MapReduce

mapred-site.xml

<property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-senior.cj.com:10020</value>
    </property>

     <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-senior.cj.com:19888</value>
    </property>

sbin/mr-jobhistory-daemon.sh stop historyserver