hadoop YARN部署并运行MapReduce测试程序

最新推荐文章于 2022-04-05 14:12:01 发布

sz_lili

最新推荐文章于 2022-04-05 14:12:01 发布

阅读量356

点赞数

CC 4.0 BY-SA版权

分类专栏： yarn 文章标签： hadoop

本文链接：https://blog.youkuaiyun.com/sz_lili/article/details/94961514

yarn 专栏收录该内容

0 篇文章

订阅专栏

本文详细介绍了如何部署Hadoop YARN，并在YARN上运行MapReduce测试程序。首先，需要配置mapred-site.xml和yarn-site.xml文件，接着启动YARN服务，然后进行MapReduce程序的测试，最后关闭YARN。MapReduce作为Hadoop的计算引擎，虽然在企业中逐渐被更简单的解决方案如Hive SQL取代，但仍不失为理解分布式计算的重要工具。

前记：
yarn:资源和作业的调度,需部署
MapReduce:hadoop计算引擎，是java开发的jar包，代码复杂，企业不用，一般用hive sql
不需要部署，运行在Yarn上
yarn部署参数配置：
1.修改mapred-site.xml文件

[hadoop@Hadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@Hadoop001 hadoop]$ vi mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>   <!--该参数是说明你的mr程序是部署在yarn上，即使用yarn调度-->
        <value>yarn</value>
    </property>
</configuration>

2.修改yarn-site.xml文件

[hadoop@Hadoop001 hadoop]$ vi yarn-site.xml 
<configuration>
   <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
   </property>
</configuration>

3.启动yarn

[hadoop@hadoop001 hadoop]$ start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop001.out
hadoop001: starting nodemanager, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop001.out
[hadoop@hadoop001 hadoop]$ jps
19648 ResourceManager
19747 NodeManager
20085 Jps
5478 SecondaryNameNode
5309 DataNode
5182 NameNode

4.测试MapReduce程序

[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir -p /example/input
19/07/07 09:35:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir -p /example/output
19/07/07 09:39:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ hdfs dfs -put /home/hadoop/text/wc.txt  /example/input
19/07/07 09:50:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ hdfs dfs -cat /example/input/wc.txt
19/07/07 09:51:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hello java
hello hadoop
hello spark
yarn hive
[hadoop@hadoop001 hadoop]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /example/input/* /example/output/1
[hadoop@hadoop001 hadoop]$ hdfs dfs -cat /example/output/1/*
19/07/07 09:54:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop	1
hello	3

hive	1
java	1
spark	1
yarn	1

5.关闭yarn

[hadoop@hadoop001 hadoop]$ stop-yarn.sh 
stopping yarn daemons
stopping resourcemanager
hadoop001: stopping nodemanager
hadoop001: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop