YARN环境搭建提交MapReduce作业

使用版本: hadoop-2.6.0-cdh5.7.0

yarn-site.xml

mapred-site.xml

启停YARN

 

cd /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

由于只有mapred-site.xml.template文件，所以cp mapred-site.xml.template mapred-site.xml

mapred-site.xml 

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

 

yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

 

启动yarn

$ sbin/stop-yarn.sh

验证是否启动成功

jps

8274 NameNode

8402 DataNode

9033 NodeManager

9370 Jps

8571 SecondaryNameNode

8892 ResourceManager

 

页面验证

http://192.168.0.66:8088/cluster

 

停止yarn

  $ sbin/stop-yarn.sh

 

在根目录下新建data文件夹，文件夹下新增hello.txt文件

文件内容为

hello world welcome

hello welcome

 

执行命令在hdfs上新建文件夹

hadoop fs -mkdir -p /input/wc

 

将hello.txt copy到hdfs文件夹下

hadoop fs -put hello.txt /input/wc

 

查看文件

hadoop fs -text /input/wc/hello.txt

 

提交mr作业到yarn上运行:wc

/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce

hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar

 

hadoop  jar /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /input/wc/hello.txt /output/wc

 

控制台打印

19/07/14 23:21:54 WARN util.NativeCodeLoader: Unable to l