配置hadoop组件yarn及案例测试

一、vi命令模式编辑配置mapred-site.xml参数
配置etc/hadoop/yarn-site.xml参数

[hadoop@hadoop001 etc]$ cd hadoop
[hadoop@hadoop001 hadoop]$ ll
#
-rw-r--r--. 1 hadoop hadoop   758 Mar 24  2016 mapred-site.xml.template
[hadoop@hadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@hadoop001 hadoop]$ vi mapred-site.xml
[hadoop@hadoop001 hadoop]$ cat mapred-site.xml

#<configuration>
    <property>
            <name>mapreduce.framework.name</name>
                    <value>yarn</value>
                        </property>
#</configuration>

[hadoop@hadoop001 hadoop]$ vi yarn-site.xml
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

二、启动yarn服务

#ResourceManager  rm 资源管理者
#NodeManager      nm 节点管理者
[hadoop@hadoop001 hadoop]$ cd ../../
[hadoop@hadoop001 hadoop]$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop001.out
hadoop001: starting nodemanager, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop001.out
[hadoop@hadoop001 hadoop]$ jps
5296 SecondaryNameNode
5010 NameNode
5107 DataNode
5940 NodeManager
5848 ResourceManager
6251 Jps

三、验证

[hadoop@hadoop001 hadoop]$ netstat -nlp|grep 5848
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 :::8088                     :::*                        LISTEN      5848/java           
tcp        0      0 :::8030                     :::*                        LISTEN      5848/java           
tcp        0      0 :::8031                     :::*                        LISTEN      5848/java           
tcp        0      0 :::8032                     :::*                        LISTEN      5848/java           
tcp        0      0 :::8033                     :::*                        LISTEN      5848/java   

四、案例展示

[hadoop@hadoop001 hadoop]$ find ./ -name '*example*.jar'
./share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.7.0.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.7.0-test-sources.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.7.0-sources.jar
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar
[hadoop@hadoop001 hadoop]$ 
[hadoop@hadoop001 hadoop]$ bin/hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar

[hadoop@hadoop001 hadoop]$  bin/hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount
Usage: wordcount <in> [<in>...] <out>
造一组词
[hadoop@hadoop001 hadoop]$ vi 1.log     
[hadoop@hadoop001 hadoop]$ vi 2.log
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir /examples/input
19/07/06 18:54:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -put *.log /examples/input
19/07/06 18:56:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
#查看vi命令编辑的1.log和2.log是否存在
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /examples/input   
19/07/06 18:57:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup         16 2019-07-06 18:56 /examples/input/1.log
-rw-r--r--   1 hadoop supergroup         38 2019-07-06 18:56 /examples/input/2.log
#执行jar包wordcount的输入输出
[hadoop@hadoop001 hadoop]$ bin/hadoop jar \
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar \
wordcount /examples/input /examples/output1
.....

[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /examples/output1
19/07/03 22:49:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2019-07-03 22:40 /examples/output1/_SUCCESS
-rw-r--r--   1 hadoop supergroup         44 2019-07-03 22:40 /examples/output1/part-r-00000
#结果输出
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /examples/output1
19/07/03 22:49:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2019-07-03 22:40 /examples/output1/_SUCCESS
-rw-r--r--   1 hadoop supergroup         44 2019-07-03 22:40 /examples/output1/part-r-00000