Hadoop(七) -- Yarn

本文详细介绍了Hadoop YARN的架构,包括ResourceManager和NodeManager的职责,以及YARN的资源调度过程。ResourceManager作为主控节点,负责资源分配、应用程序管理和监控,而NodeManager管理单个节点的资源。YARN采用Container作为资源调度的最小单位。资源调度过程中,MRAppMaster负责申请和跟踪任务资源,确保任务的执行。文章还概述了Job提交的流程,从客户端提交到任务执行的完整步骤。

一、Yarn概述

资源调度器,负责计算程序的资源调度。

Yarn采用主从架构,主节点RecourceManager,从节点NodeManager。

1. ResourceManager

        ResourceManager是基于应用程序对集群资源的需求进行调度的YARN集群主控节点,负责协调和管理整个集群(所有nodemanager的资源),相应用户提交的不同类型应用程序的解析,调度,监控等工作。ResourceManager为每个Application启动一个MRAppMaster,并且MRAppMaster分散在各个NodeManager节点。

        作用:

                1)处理客户端请求

                2)启动并监控MRAppMaster

                3)监控NodeManager

                4)资源的分配和调度

stu@stu-pc:~$ find / -name "hadoop" -type d 2>/dev/null | grep -i hadoop/home/stu/hadoop/usr/local/hadoop/usr/local/hadoop/etc/hadoop/usr/local/hadoop/share/hadoop/usr/local/hadoop/share/doc/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-app/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-uploader/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/org/apache/hadoop/lib/service/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-auth/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-api/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-registry/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-client/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/api/org/apache/hadoop/lib/service/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs-client/build/source/hadoop-hdfs-project/hadoop-hdfs-client/target/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-common/build/source/hadoop-common-project/hadoop-common/target/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs-rbf/build/source/hadoop-hdfs-project/hadoop-hdfs-rbf/target/api/org/apache/hadoop/usr/local/hadoop/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/build/source/hadoop-hdfs-project/hadoop-hdfs/target/api/org/apache/hadoop/usr/local/hive/lib/php/packages/serde/org/apache/hadoop/usr/local/hive/apache-hive-3.1.2-bin/lib/php/packages/serde/org/apache/hadoop/tmp/hadoop/tmp/hadoop-unjar8342885896351558271/org/apache/hadoop/tmp/hadoop-unjar6484726597599168525/org/apache/hadoopstu@stu-pc:~$ ls /usr/local/hadoopbin data etc include lib libexec LICENSE.txt logs NOTICE.txt README.txt sbin share tmpstu@stu-pc:~$ ls /opt/hadoopls: 无法访问'/opt/hadoop': 没有那个文件目录stu@stu-pc:~$
最新发布
12-09
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值